Nov 02 19:25:12.225405 ip-10-0-132-174 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 19:25:12.225416 ip-10-0-132-174 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 19:25:12.225423 ip-10-0-132-174 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 19:25:12.225647 ip-10-0-132-174 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 19:25:22.417179 ip-10-0-132-174 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 19:25:22.417195 ip-10-0-132-174 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 018a44d2dc6b49a9b4d2f99beafbcc1f -- Nov 02 19:27:31.407627 ip-10-0-132-174 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:27:31.815685 ip-10-0-132-174 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:27:31.815685 ip-10-0-132-174 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:27:31.815685 ip-10-0-132-174 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:27:31.815685 ip-10-0-132-174 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:27:31.822311 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822202 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.822311 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822291 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822319 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822324 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822328 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822333 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822337 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822342 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822345 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822349 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822353 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822357 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822361 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822365 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822369 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822373 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822377 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822381 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822384 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822388 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822392 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.822493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822396 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822400 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822405 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822409 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822413 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822419 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822423 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822427 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822432 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822436 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822448 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822453 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822457 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822462 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822466 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822471 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822475 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822479 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822484 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822488 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.823317 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822492 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822496 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822500 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822505 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822510 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822514 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822518 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822523 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822527 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822532 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822536 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822540 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822545 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822549 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822554 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822562 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822567 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822571 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822575 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.824051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822580 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822584 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822588 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822593 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822597 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822601 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822606 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822610 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822616 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822620 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822624 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822628 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822636 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822643 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822648 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822652 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822656 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822661 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822665 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822669 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.824542 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822673 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822678 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822682 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822693 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822697 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822702 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822706 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822712 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822716 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822720 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822725 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822729 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822733 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822738 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822744 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822752 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822757 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822764 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.825173 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822771 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822776 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822781 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822785 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822790 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822794 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.822798 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825086 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825098 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825103 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825108 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825113 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825118 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825122 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825126 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825130 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825135 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825142 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825147 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825151 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.826051 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825155 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825161 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825165 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825170 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825174 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825178 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825182 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825186 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825190 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825194 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825198 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825202 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825207 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825211 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825216 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825220 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825224 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825229 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825234 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825238 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.826956 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825242 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825246 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825251 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825255 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825263 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825267 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825271 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825275 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825279 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825285 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825290 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825315 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825322 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825326 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825331 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825335 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825339 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825343 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825348 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.827527 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825352 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825357 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825361 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825365 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825369 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825373 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825378 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825385 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825390 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825395 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825401 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825405 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825409 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825414 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825419 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825423 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825427 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825431 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.828014 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825436 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825440 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825444 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825448 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825453 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825458 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825463 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825468 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825472 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825477 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825481 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825486 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825490 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825494 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825498 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825503 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825511 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825515 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825520 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.828861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825524 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825528 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825532 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825536 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825541 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825545 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825549 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825553 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825557 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825561 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825566 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825569 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825574 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825580 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.825584 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826446 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826456 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826463 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826468 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.829754 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826473 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826477 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826483 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826488 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826492 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826496 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826501 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826505 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826509 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826513 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826517 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826522 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826526 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826530 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826535 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826541 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826548 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826553 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.830521 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826557 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826561 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826566 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826573 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826577 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826581 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826586 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826590 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826594 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826598 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826602 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826606 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826611 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826616 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826620 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826626 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826630 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826634 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826639 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826643 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.830954 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826648 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826651 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826660 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826664 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826668 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826672 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826676 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826680 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826685 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826690 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826694 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826699 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826703 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826708 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826716 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826722 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826727 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826732 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826736 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.831700 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826740 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826745 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826749 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826753 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826757 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826761 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826770 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826775 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826780 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826784 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826788 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826792 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826796 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826801 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826805 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826809 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826813 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826817 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826822 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826826 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.832193 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826830 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826834 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826839 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826843 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826847 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826852 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826856 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826861 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826866 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826870 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826874 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826879 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826883 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826887 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826892 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826896 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826902 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826906 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826910 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826915 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.832703 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826920 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826924 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826929 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.826933 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828040 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828058 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828070 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828077 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828084 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828089 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828097 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828103 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828109 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828113 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828119 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828124 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828130 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828135 2578 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828140 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828146 2578 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828151 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828156 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828162 2578 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828166 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:27:31.833268 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828171 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828177 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828184 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828192 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828197 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828202 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828207 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828212 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828218 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828223 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828229 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828233 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828241 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828246 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828250 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828255 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828260 2578 flags.go:64] FLAG: --enable-server="true" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828265 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828274 2578 flags.go:64] FLAG: --event-burst="100" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828280 2578 flags.go:64] FLAG: --event-qps="50" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828284 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828289 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828310 2578 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828318 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:27:31.833862 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828323 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828328 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828333 2578 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828340 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828345 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828350 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828355 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828359 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828364 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828369 2578 flags.go:64] FLAG: --feature-gates="" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828376 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828381 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828386 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828391 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828396 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828401 2578 flags.go:64] FLAG: --help="false" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828405 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828412 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828417 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828422 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828427 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828433 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828438 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:27:31.834540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828442 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828447 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828451 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828457 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828463 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828467 2578 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828472 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828477 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828482 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828486 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828491 2578 flags.go:64] FLAG: --lock-file="" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828495 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828502 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828507 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828517 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828522 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828527 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828532 2578 flags.go:64] FLAG: --logging-format="text" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828536 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828542 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828547 2578 flags.go:64] FLAG: --manifest-url="" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828552 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828559 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828564 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828570 2578 flags.go:64] FLAG: --max-pods="110" Nov 02 19:27:31.835088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828575 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828580 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828585 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828591 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828595 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828600 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828605 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828617 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828621 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828627 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828632 2578 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828636 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828641 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828646 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828651 2578 flags.go:64] FLAG: --port="10250" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828655 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828660 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0010e02df6a54c2de" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828665 2578 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828670 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828676 2578 flags.go:64] FLAG: --register-node="true" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828681 2578 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828688 2578 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828692 2578 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828697 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828702 2578 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828709 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:27:31.835711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828714 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828719 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828724 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828728 2578 flags.go:64] FLAG: --runonce="false" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828733 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828738 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828743 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828748 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828753 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828759 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828791 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828799 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828804 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828810 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828814 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828819 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828824 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828829 2578 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828833 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828842 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828847 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828851 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828858 2578 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828867 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828871 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:27:31.836416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828876 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828880 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828885 2578 flags.go:64] FLAG: --v="2" Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828892 2578 flags.go:64] FLAG: --version="false" Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828899 2578 flags.go:64] FLAG: --vmodule="" Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828906 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.828912 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829132 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829139 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829145 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829150 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829155 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829160 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829165 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829169 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829174 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829178 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829183 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829188 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829192 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829196 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.837009 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829200 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829204 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829209 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829213 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829218 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829222 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829227 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829231 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829235 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829242 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829246 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829250 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829254 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829259 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829263 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829267 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829271 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829276 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829283 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829288 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.837524 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829293 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829313 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829317 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829321 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829325 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829330 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829334 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829338 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829342 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829348 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829352 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829356 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829360 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829365 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829368 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829373 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829377 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829381 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829386 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829391 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.838001 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829395 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829401 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829405 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829410 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829414 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829419 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829423 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829428 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829432 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829436 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829440 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829445 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829451 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829457 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829462 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829467 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829472 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829477 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829481 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.838498 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829486 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829490 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829500 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829504 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829509 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829513 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829517 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829522 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829526 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829530 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829534 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829538 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829543 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829549 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829553 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829558 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829562 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829566 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829571 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.839005 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829576 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829580 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829584 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829589 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829594 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829598 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829603 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829607 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829611 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829615 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829619 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829623 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829627 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.829671 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829745 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829752 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.839504 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829757 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829761 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829767 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829772 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829777 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829781 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829785 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829789 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829794 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829799 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829803 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829811 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829815 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829819 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829824 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829828 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829832 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829837 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829841 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.839973 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829846 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829850 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829854 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829858 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829862 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829867 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829871 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829875 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829879 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829884 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829889 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829893 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829897 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829902 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829908 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829912 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829916 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829920 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829924 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829928 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.840505 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829933 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829937 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829941 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829945 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829953 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829957 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829962 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829966 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829970 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829975 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829979 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829983 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829987 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829992 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.829996 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830000 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830004 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830009 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830013 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830019 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.840978 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830023 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830027 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830032 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830036 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830040 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830044 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830048 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830053 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830057 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830061 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830066 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830070 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830074 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830078 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830082 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830089 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830099 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830105 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830109 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830114 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.841460 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830118 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830122 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830126 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830131 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830135 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830139 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830143 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830148 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830152 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830156 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830160 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830165 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830169 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830173 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830178 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830184 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830188 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830192 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830196 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.841934 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830201 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830206 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830210 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830214 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830218 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.830223 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.837828 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.837843 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.838641 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.838649 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838709 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838715 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838719 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838722 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838725 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838728 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838730 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838733 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838736 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.842406 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838739 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838742 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838744 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838747 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838749 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838752 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838755 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838757 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838760 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838762 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838769 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838772 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838774 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838777 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838781 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838784 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838787 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838789 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838792 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.842861 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838795 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838799 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838801 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838804 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838807 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838809 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838812 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838815 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838817 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838820 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838823 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838825 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838828 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838830 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838833 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838836 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838838 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838841 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838844 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.843338 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838846 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838849 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838851 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838854 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838857 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838859 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838862 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838867 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838870 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838873 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838876 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838878 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838881 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838884 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838886 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838889 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838892 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838894 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838897 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.843785 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838899 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838902 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838904 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838907 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838909 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838912 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838914 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838917 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838920 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838923 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838927 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838930 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838932 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838935 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838938 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838940 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838942 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838945 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838948 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838951 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.844316 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838953 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838957 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838962 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838965 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838967 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838970 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838973 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838976 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838979 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838981 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838984 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838987 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838989 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838991 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838994 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838997 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.838999 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839002 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.844803 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839004 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.839032 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839178 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839182 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839185 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839188 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839191 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839194 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839196 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839199 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839201 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839204 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839207 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839209 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839212 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839214 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.845225 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839217 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839220 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839223 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839225 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839228 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839231 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839233 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839236 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839241 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839244 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839246 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839249 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839252 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839254 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839257 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839259 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839262 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839264 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839267 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.845616 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839269 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839272 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839274 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839277 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839279 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839282 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839284 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839287 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839290 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839292 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839308 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839311 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839314 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839317 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839320 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839323 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839326 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839329 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839332 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839334 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.846081 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839337 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839339 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839342 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839345 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839347 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839350 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839352 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839355 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839357 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839360 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839362 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839365 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839368 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839370 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839373 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839375 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839378 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839381 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839383 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.846584 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839386 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839388 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839391 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839393 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839396 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839398 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839401 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839403 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839406 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839409 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839412 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839414 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839417 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839419 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839422 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839424 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839426 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839429 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839432 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.847047 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839436 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839440 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839443 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839447 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839450 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839452 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839455 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839458 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839462 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839466 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839470 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839473 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839475 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839478 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.839501 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:27:31.847595 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839539 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839544 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839546 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839549 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839551 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839554 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839557 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839561 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839565 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839568 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839571 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839574 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839576 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839579 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839581 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839584 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839586 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839589 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839591 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839594 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:27:31.847942 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839596 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839599 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839601 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839604 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839606 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839609 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839611 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839614 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839616 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839619 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839622 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839624 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839627 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839629 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839631 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839634 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839637 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839639 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839642 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839644 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:27:31.848432 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839647 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839650 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839652 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839655 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839657 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839660 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839662 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839665 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839667 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839669 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839672 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839674 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839680 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839682 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839684 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839687 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839689 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839692 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839694 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839697 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:27:31.848896 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839700 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839702 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839705 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839708 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839710 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839713 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839716 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839718 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839721 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839724 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839727 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839729 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839732 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839735 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839737 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839740 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839742 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839745 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839748 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839750 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:27:31.849380 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839753 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839755 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839758 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839760 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839765 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839767 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839770 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839772 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839774 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839777 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839782 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839784 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839787 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839789 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839792 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839794 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839797 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839799 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839802 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:27:31.849854 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839804 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839807 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839809 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839814 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839817 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:31.839822 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.839875 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.843207 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.844334 2578 server.go:1014] "Starting client certificate rotation" Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.844427 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.844465 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.846665 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:27:31.850339 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.848210 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:27:31.859654 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.859636 2578 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:27:31.863600 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.863588 2578 log.go:25] "Validated CRI v1 image API" Nov 02 19:27:31.864461 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.864438 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:27:31.870378 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.870356 2578 fs.go:135] Filesystem UUIDs: map[1c016816-0d4b-4289-977d-b35f16ba7215:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b8ab3d44-23a2-4bc4-8c96-e3f75df8e0b4:/dev/nvme0n1p4] Nov 02 19:27:31.870446 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.870375 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 19:27:31.876878 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.876760 2578 manager.go:217] Machine: {Timestamp:2026-11-02 19:27:31.874541206 +0000 UTC m=+0.369919719 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100118 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ada901e446db8cf6ce7dfb87c149 SystemUUID:ec23ada9-01e4-46db-8cf6-ce7dfb87c149 BootID:018a44d2-dc6b-49a9-b4d2-f99beafbcc1f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e3:4e:4c:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e3:4e:4c:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:20:ae:4b:3d:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:27:31.876878 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.876877 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:27:31.876985 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.876963 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:27:31.878236 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.878194 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:27:31.878437 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.878242 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-174.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:27:31.878493 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.878440 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:27:31.878493 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.878449 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:27:31.878493 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.878486 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:27:31.878577 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.878503 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:27:31.879743 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.879732 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:27:31.884477 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.884465 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:27:31.884592 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.884583 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:27:31.885416 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.885407 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:27:31.885455 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.885420 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:27:31.885455 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.885434 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:27:31.885455 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.885443 2578 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:27:31.885455 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.885451 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:27:31.885831 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.885805 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:27:31.888308 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.888263 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:27:31.894921 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.894908 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:27:31.895827 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.895813 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:27:31.895864 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.895845 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:27:31.896007 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.895996 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:27:31.896046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896012 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:27:31.896046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896018 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:27:31.896046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896025 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:27:31.896046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896030 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:27:31.896046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896036 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:27:31.896795 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896786 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:27:31.896823 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896796 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:27:31.896823 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896803 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:27:31.896823 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896810 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:27:31.896823 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896818 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:27:31.896996 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.896828 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:27:31.897581 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.897572 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:27:31.897632 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.897585 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:27:31.901495 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.901481 2578 server.go:1263] "Started kubelet" Nov 02 19:27:31.901606 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.901580 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:27:31.901654 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.901573 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:27:31.901691 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.901657 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:27:31.902257 ip-10-0-132-174 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:27:31.903254 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.903239 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:27:31.909492 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.909476 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:27:31.909931 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:31.909908 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 19:27:31.914259 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.914239 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:27:31.915612 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.915598 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:27:31.915735 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:31.915715 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:31.916337 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.916318 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:27:31.916429 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.916388 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:27:31.916429 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.916402 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:27:31.916545 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.916515 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:27:31.916545 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.916527 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:27:31.917866 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.917844 2578 factory.go:55] Registering systemd factory Nov 02 19:27:31.917866 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.917861 2578 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:27:31.918060 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.918048 2578 factory.go:153] Registering CRI-O factory Nov 02 19:27:31.918103 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.918063 2578 factory.go:223] Registration of the crio container factory successfully Nov 02 19:27:31.918103 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.918101 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:27:31.918157 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.918121 2578 factory.go:103] Registering Raw factory Nov 02 19:27:31.918157 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.918134 2578 manager.go:1199] Started watching for new ooms in manager Nov 02 19:27:31.919181 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.919168 2578 manager.go:319] Starting recovery of all containers Nov 02 19:27:31.929288 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.929274 2578 manager.go:324] Recovery completed Nov 02 19:27:31.933817 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.933802 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:31.935614 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.935586 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:31.935686 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.935620 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:31.935686 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.935634 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:31.936146 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.936132 2578 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:27:31.936146 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.936144 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:27:31.936251 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.936163 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:27:31.936251 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.936167 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:31.936855 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.936842 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:31.936912 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.936866 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:31.936912 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.936875 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:31.938375 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.938357 2578 policy_none.go:50] "Start" Nov 02 19:27:31.938448 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.938380 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:27:31.938448 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.938393 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:27:31.940384 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.940369 2578 policy_none.go:44] "Start" Nov 02 19:27:31.968970 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.968935 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.973555 2578 manager.go:341] "Starting Device Plugin manager" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:31.973598 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.973608 2578 server.go:87] "Starting device plugin registration server" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.973845 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.973854 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.973938 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.974027 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.974036 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:31.974532 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:31.974586 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.987411 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.988515 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.988535 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:31.988556 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:27:31.993602 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:31.988589 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 19:27:32.032210 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.032186 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-174.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 19:27:32.032339 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.032225 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-174.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 19:27:32.033292 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.032193 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-174.us-west-2.compute.internal.18e4510947ccbb60 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:27:31.9014552 +0000 UTC m=+0.396833712,LastTimestamp:2026-11-02 19:27:31.9014552 +0000 UTC m=+0.396833712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-174.us-west-2.compute.internal,}" Nov 02 19:27:32.074417 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.074357 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:32.075459 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.075439 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:32.075558 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.075470 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:32.075558 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.075488 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:32.075558 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.075520 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.090156 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.090138 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal"] Nov 02 19:27:32.090229 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.090190 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:32.091118 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.091098 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:32.091208 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.091129 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:32.091208 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.091139 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:32.093371 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.093359 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:32.093514 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.093499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.093558 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.093534 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:32.094183 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.094164 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:32.094183 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.094170 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:32.094332 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.094192 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:32.094332 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.094198 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:32.094332 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.094204 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:32.094332 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.094240 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:32.096331 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.096313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.096434 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.096342 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:32.097030 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.097017 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:32.097089 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.097038 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:32.097089 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.097049 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:32.098854 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.098830 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 19:27:32.098854 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.098834 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 19:27:32.098998 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.098838 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-174.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 19:27:32.099088 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.098818 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-174.us-west-2.compute.internal.18e4510949d5d027 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-174.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:27:31.935604775 +0000 UTC m=+0.430983287,LastTimestamp:2026-11-02 19:27:31.935604775 +0000 UTC m=+0.430983287,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-174.us-west-2.compute.internal,}" Nov 02 19:27:32.101778 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.101756 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:27:32.101860 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.101844 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwnmv" Nov 02 19:27:32.117784 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.117754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.117784 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.117786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.117943 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.117807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.123993 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.123973 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.126008 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.125991 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.177844 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.177813 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:32.187229 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.187206 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.187357 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.187234 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-174.us-west-2.compute.internal\": node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.218497 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.218463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.218591 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.218502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.218591 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.218529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.218591 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.218581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.218682 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.218582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.218682 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.218581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.289124 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.289092 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.389785 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.389721 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.428986 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.428961 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.431054 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.431030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:32.490294 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.490148 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.590798 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.590764 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.691309 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.691222 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.792228 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.792201 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.844273 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.844250 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 19:27:32.844668 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:32.844404 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:27:32.844668 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.844478 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a15d747bf66324e00b1608078aae7d90-1121cf0d51ffc33d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-174.us-west-2.compute.internal.18e4510949d65204\": read tcp 10.0.132.174:48154->3.151.92.97:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-174.us-west-2.compute.internal.18e4510949d65204 default 4865 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-174.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:27:31 +0000 UTC,LastTimestamp:2026-11-02 19:27:32.091144496 +0000 UTC m=+0.586523008,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-174.us-west-2.compute.internal,}" Nov 02 19:27:32.892661 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.892629 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:32.992884 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:32.992816 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:33.075279 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.075251 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:27:33.076150 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.076135 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:27:33.076260 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.076167 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:27:33.076260 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.076180 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:27:33.093412 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:33.093392 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:33.107536 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.107502 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:22:31 +0000 UTC" deadline="2028-07-19 16:45:22.55484376 +0000 UTC" Nov 02 19:27:33.107536 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.107533 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14997h17m49.447313786s" Nov 02 19:27:33.194196 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:33.194167 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:27:33.201943 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.201923 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:33.203105 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.203092 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:33.216172 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.216148 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:33.250095 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.250030 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:33.274230 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.274202 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:27:33.275085 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.275070 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:27:33.326323 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.326282 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:27:33.399563 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:33.399538 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747e2de570be6f488466605f00369eb2.slice/crio-32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240 WatchSource:0}: Error finding container 32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240: Status 404 returned error can't find the container with id 32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240 Nov 02 19:27:33.408452 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.408437 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:27:33.413694 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:33.413669 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac4092c55873e071f156a04a17c98a5.slice/crio-8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a WatchSource:0}: Error finding container 8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a: Status 404 returned error can't find the container with id 8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a Nov 02 19:27:33.887358 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.887332 2578 apiserver.go:52] "Watching apiserver" Nov 02 19:27:33.914851 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.914825 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 19:27:33.993799 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.993754 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a"} Nov 02 19:27:33.994825 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:33.994800 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240"} Nov 02 19:27:34.034265 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.034235 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:27:34.034551 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.034534 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xfpnh","openshift-network-operator/iptables-alerter-jdlx5","openshift-ovn-kubernetes/ovnkube-node-8thp7","openshift-multus/multus-additional-cni-plugins-zs222","openshift-multus/network-metrics-daemon-7hnrd","openshift-network-diagnostics/network-check-target-wmg7k","kube-system/global-pull-secret-syncer-2dw9d","openshift-image-registry/node-ca-j585w","openshift-multus/multus-789jj","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7","openshift-cluster-node-tuning-operator/tuned-rglh2","openshift-dns/node-resolver-s4b6j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal"] Nov 02 19:27:34.038131 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.038109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.038201 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.038184 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:34.040270 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.040257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.042483 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.042464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.044719 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.044699 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.046895 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.046871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:34.046973 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.046936 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:34.050049 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.050034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:34.050119 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.050083 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:34.052482 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.052342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-789jj" Nov 02 19:27:34.054660 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.054644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.056909 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.056895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.059149 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.059133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.061367 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.061349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.063697 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.063681 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.092648 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.092630 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:27:34.092732 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.092657 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:27:34.092732 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.092704 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxdrr\"" Nov 02 19:27:34.117382 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.117361 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:27:34.130872 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.130854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:34.130965 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.130880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-multus\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.130965 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.130896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.130965 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.130912 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwfh\" (UniqueName: \"kubernetes.io/projected/9e81006a-11bf-44aa-8561-1de0236016c8-kube-api-access-6cwfh\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.130965 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.130928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-multus-certs\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.131169 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.130978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.131169 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-systemd\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.131169 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-netns\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.131169 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-bin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.131169 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-var-lib-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzg5\" (UniqueName: \"kubernetes.io/projected/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kube-api-access-rgzg5\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6z6h\" (UniqueName: \"kubernetes.io/projected/52b4ca7a-3abb-4c17-a460-51f0b45cadca-kube-api-access-t6z6h\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9gqw\" (UniqueName: \"kubernetes.io/projected/c4a482a0-025f-4bd6-8229-aa6971510e12-kube-api-access-q9gqw\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-netns\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-bin\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-kubelet\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.131379 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-etc-kubernetes\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d14e549d-addd-40f6-b6af-12582e1a028b-agent-certs\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-hostroot\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4pq\" (UniqueName: \"kubernetes.io/projected/9e7785c4-f87a-4969-9d54-1aafb3c2e641-kube-api-access-dl4pq\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4a482a0-025f-4bd6-8229-aa6971510e12-iptables-alerter-script\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-kubelet-config\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-etc-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-host\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-serviceca\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgdq\" (UniqueName: \"kubernetes.io/projected/9a9afd80-1122-46ad-8082-77f3b20738e9-kube-api-access-xkgdq\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-config\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-script-lib\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.131738 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mr9n\" (UniqueName: \"kubernetes.io/projected/3fc1dba1-1e3a-4639-b2da-8277acf64efb-kube-api-access-6mr9n\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-system-cni-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-socket-dir-parent\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-conf-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-kubelet\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-device-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-kubernetes\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cni-binary-copy\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.131997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-dbus\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-systemd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-netd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqv5g\" (UniqueName: \"kubernetes.io/projected/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-kube-api-access-wqv5g\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-socket-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-var-lib-kubelet\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-cnibin\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.132232 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d14e549d-addd-40f6-b6af-12582e1a028b-konnectivity-ca\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-node-log\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-modprobe-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-run\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-sys\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-system-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cnibin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132409 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-sys-fs\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-host\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-tuned\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b4ca7a-3abb-4c17-a460-51f0b45cadca-tmp-dir\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-systemd-units\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-slash\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-env-overrides\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-registration-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.132910 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysconfig\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-daemon-config\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-conf\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-lib-modules\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-tmp\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9qr\" (UniqueName: \"kubernetes.io/projected/3a889062-1b3c-442a-860d-64e9209a6acf-kube-api-access-pr9qr\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-log-socket\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e81006a-11bf-44aa-8561-1de0236016c8-ovn-node-metrics-cert\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52b4ca7a-3abb-4c17-a460-51f0b45cadca-hosts-file\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.132981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.133005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-k8s-cni-cncf-io\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.133040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a482a0-025f-4bd6-8229-aa6971510e12-host-slash\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.133066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-ovn\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.133567 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.133093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.138143 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.138106 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:27:34.138265 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.138253 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bb8t\"" Nov 02 19:27:34.138321 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.138253 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lnn7q\"" Nov 02 19:27:34.138597 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.138581 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:27:34.138660 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.138638 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c9dm8" Nov 02 19:27:34.138939 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.138922 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:27:34.139015 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139001 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:27:34.139091 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139075 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4nhx\"" Nov 02 19:27:34.139133 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139078 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:27:34.139278 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139262 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:27:34.139278 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139267 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139338 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139354 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139357 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139389 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gsgfx\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139408 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139415 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139415 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139430 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139442 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:27:34.139496 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.139444 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nnqx8\"" Nov 02 19:27:34.142985 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.142971 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:27:34.143064 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.143052 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:27:34.143064 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.143060 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:27:34.143137 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.143056 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:27:34.188714 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.188686 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:27:34.188714 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.188686 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:27:34.188892 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.188717 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:27:34.188892 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.188751 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:27:34.188892 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.188818 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t85s\"" Nov 02 19:27:34.189521 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.189510 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:27:34.189562 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.189519 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:27:34.189598 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.189566 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bg56f\"" Nov 02 19:27:34.189646 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.189633 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sp8wk\"" Nov 02 19:27:34.189715 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.189698 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:27:34.189762 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.189726 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:27:34.233989 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.233963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b4ca7a-3abb-4c17-a460-51f0b45cadca-tmp-dir\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.233989 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.233992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-systemd-units\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-slash\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-env-overrides\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-slash\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-systemd-units\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-registration-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-registration-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysconfig\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.234218 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-daemon-config\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysconfig\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-conf\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-lib-modules\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-tmp\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b4ca7a-3abb-4c17-a460-51f0b45cadca-tmp-dir\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9qr\" (UniqueName: \"kubernetes.io/projected/3a889062-1b3c-442a-860d-64e9209a6acf-kube-api-access-pr9qr\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.234452 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-log-socket\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e81006a-11bf-44aa-8561-1de0236016c8-ovn-node-metrics-cert\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-conf\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52b4ca7a-3abb-4c17-a460-51f0b45cadca-hosts-file\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234512 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.234532 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret podName:fea7ce1f-d9c4-40b4-a160-0f7788c416a6 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:34.734511484 +0000 UTC m=+3.229890004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret") pod "global-pull-secret-syncer-2dw9d" (UID: "fea7ce1f-d9c4-40b4-a160-0f7788c416a6") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:34.235338 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-env-overrides\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-log-socket\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-lib-modules\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234554 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52b4ca7a-3abb-4c17-a460-51f0b45cadca-hosts-file\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-k8s-cni-cncf-io\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a482a0-025f-4bd6-8229-aa6971510e12-host-slash\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-ovn\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234729 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-k8s-cni-cncf-io\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-multus\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-ovn\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a482a0-025f-4bd6-8229-aa6971510e12-host-slash\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234841 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwfh\" (UniqueName: \"kubernetes.io/projected/9e81006a-11bf-44aa-8561-1de0236016c8-kube-api-access-6cwfh\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.235870 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-daemon-config\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-multus-certs\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-multus\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.234905 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234830 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234934 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-multus-certs\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234942 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234953 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.234965 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:27:34.734951152 +0000 UTC m=+3.230329670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.234996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-systemd\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-netns\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-bin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235084 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-netns\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-systemd\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-bin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-var-lib-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.236540 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-var-lib-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzg5\" (UniqueName: \"kubernetes.io/projected/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kube-api-access-rgzg5\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235201 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235200 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6z6h\" (UniqueName: \"kubernetes.io/projected/52b4ca7a-3abb-4c17-a460-51f0b45cadca-kube-api-access-t6z6h\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9gqw\" (UniqueName: \"kubernetes.io/projected/c4a482a0-025f-4bd6-8229-aa6971510e12-kube-api-access-q9gqw\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-netns\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-bin\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-kubelet\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-etc-kubernetes\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-netns\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d14e549d-addd-40f6-b6af-12582e1a028b-agent-certs\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-hostroot\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4pq\" (UniqueName: \"kubernetes.io/projected/9e7785c4-f87a-4969-9d54-1aafb3c2e641-kube-api-access-dl4pq\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.237385 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4a482a0-025f-4bd6-8229-aa6971510e12-iptables-alerter-script\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-kubelet-config\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-etc-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-host\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-serviceca\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgdq\" (UniqueName: \"kubernetes.io/projected/9a9afd80-1122-46ad-8082-77f3b20738e9-kube-api-access-xkgdq\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-config\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-script-lib\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mr9n\" (UniqueName: \"kubernetes.io/projected/3fc1dba1-1e3a-4639-b2da-8277acf64efb-kube-api-access-6mr9n\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-etc-kubernetes\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-system-cni-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-socket-dir-parent\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-hostroot\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-conf-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-kubelet\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-bin\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-device-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235953 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-etc-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-kubernetes\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.235994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cni-binary-copy\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-dbus\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-systemd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-netd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqv5g\" (UniqueName: \"kubernetes.io/projected/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-kube-api-access-wqv5g\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-socket-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-var-lib-kubelet\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-cnibin\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-kubelet\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d14e549d-addd-40f6-b6af-12582e1a028b-konnectivity-ca\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-kubelet-config\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-node-log\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-modprobe-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.238993 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-run\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-sys\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-system-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cnibin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-config\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-sys-fs\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-host\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236510 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4a482a0-025f-4bd6-8229-aa6971510e12-iptables-alerter-script\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-tuned\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236524 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236293 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-node-log\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-sys-fs\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236625 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-systemd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236630 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-run\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236667 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-sys\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-dbus\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236738 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-script-lib\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.239589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d14e549d-addd-40f6-b6af-12582e1a028b-konnectivity-ca\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cnibin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236827 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-host\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236837 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-socket-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-host\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-device-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236985 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-kubelet\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.236995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-netd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-serviceca\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-system-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-cnibin\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-var-lib-kubelet\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-conf-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-kubernetes\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-socket-dir-parent\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-modprobe-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237131 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-system-cni-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.240072 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cni-binary-copy\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.240569 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.237402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.240569 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.238749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e81006a-11bf-44aa-8561-1de0236016c8-ovn-node-metrics-cert\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.240569 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.238751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-tmp\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.240569 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.238975 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-tuned\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.240569 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.239409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d14e549d-addd-40f6-b6af-12582e1a028b-agent-certs\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.289922 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.289899 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:34.289922 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.289921 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:34.290060 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.289931 2578 projected.go:196] Error preparing data for projected volume kube-api-access-z8qhm for pod openshift-network-diagnostics/network-check-target-wmg7k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:34.290060 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.289991 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm podName:1fdecbd0-dcd5-42f0-85ce-57b32e400992 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:34.789977385 +0000 UTC m=+3.285355913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z8qhm" (UniqueName: "kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm") pod "network-check-target-wmg7k" (UID: "1fdecbd0-dcd5-42f0-85ce-57b32e400992") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:34.294613 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.292824 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9gqw\" (UniqueName: \"kubernetes.io/projected/c4a482a0-025f-4bd6-8229-aa6971510e12-kube-api-access-q9gqw\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.294613 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.293000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9qr\" (UniqueName: \"kubernetes.io/projected/3a889062-1b3c-442a-860d-64e9209a6acf-kube-api-access-pr9qr\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:34.294613 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.293382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4pq\" (UniqueName: \"kubernetes.io/projected/9e7785c4-f87a-4969-9d54-1aafb3c2e641-kube-api-access-dl4pq\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:27:34.294613 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.293573 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwfh\" (UniqueName: \"kubernetes.io/projected/9e81006a-11bf-44aa-8561-1de0236016c8-kube-api-access-6cwfh\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.294613 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.294513 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgdq\" (UniqueName: \"kubernetes.io/projected/9a9afd80-1122-46ad-8082-77f3b20738e9-kube-api-access-xkgdq\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.294829 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.294661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6z6h\" (UniqueName: \"kubernetes.io/projected/52b4ca7a-3abb-4c17-a460-51f0b45cadca-kube-api-access-t6z6h\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.294863 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.294827 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqv5g\" (UniqueName: \"kubernetes.io/projected/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-kube-api-access-wqv5g\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.294863 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.294855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mr9n\" (UniqueName: \"kubernetes.io/projected/3fc1dba1-1e3a-4639-b2da-8277acf64efb-kube-api-access-6mr9n\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.294968 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.294945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzg5\" (UniqueName: \"kubernetes.io/projected/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kube-api-access-rgzg5\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.352006 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.351980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:27:34.363493 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.363464 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b4ca7a_3abb_4c17_a460_51f0b45cadca.slice/crio-fccae350b6edf37351f93eaa658b8873d4e6f1a8e850b6a0b66ae8b19472e4d2 WatchSource:0}: Error finding container fccae350b6edf37351f93eaa658b8873d4e6f1a8e850b6a0b66ae8b19472e4d2: Status 404 returned error can't find the container with id fccae350b6edf37351f93eaa658b8873d4e6f1a8e850b6a0b66ae8b19472e4d2 Nov 02 19:27:34.375228 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.375207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-789jj" Nov 02 19:27:34.381412 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.381394 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:34.383685 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.383658 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7785c4_f87a_4969_9d54_1aafb3c2e641.slice/crio-50c6edd275cac77e2abb2a93709a893bdb350a36d624d0cc3f23589c6cab21fc WatchSource:0}: Error finding container 50c6edd275cac77e2abb2a93709a893bdb350a36d624d0cc3f23589c6cab21fc: Status 404 returned error can't find the container with id 50c6edd275cac77e2abb2a93709a893bdb350a36d624d0cc3f23589c6cab21fc Nov 02 19:27:34.384942 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.384925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:27:34.387894 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.387705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:34.390426 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.390392 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14e549d_addd_40f6_b6af_12582e1a028b.slice/crio-4e2b0c34247ff094e999c42aedf01d2c017c032ed54df4e40ecc6e722f59cca7 WatchSource:0}: Error finding container 4e2b0c34247ff094e999c42aedf01d2c017c032ed54df4e40ecc6e722f59cca7: Status 404 returned error can't find the container with id 4e2b0c34247ff094e999c42aedf01d2c017c032ed54df4e40ecc6e722f59cca7 Nov 02 19:27:34.391465 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.391447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j585w" Nov 02 19:27:34.396161 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.395867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:27:34.399382 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.399358 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a482a0_025f_4bd6_8229_aa6971510e12.slice/crio-671c112c9149b2bd8db675a1c244110b2d1173ed435e64b6a8ba1b33ac5d2ac7 WatchSource:0}: Error finding container 671c112c9149b2bd8db675a1c244110b2d1173ed435e64b6a8ba1b33ac5d2ac7: Status 404 returned error can't find the container with id 671c112c9149b2bd8db675a1c244110b2d1173ed435e64b6a8ba1b33ac5d2ac7 Nov 02 19:27:34.407340 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.406439 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:27:34.407340 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.406610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:27:34.407340 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.407108 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e81006a_11bf_44aa_8561_1de0236016c8.slice/crio-a2ef35a9e954227445b49998d78567e5aba76885a034f4cc2c45e79fef076d48 WatchSource:0}: Error finding container a2ef35a9e954227445b49998d78567e5aba76885a034f4cc2c45e79fef076d48: Status 404 returned error can't find the container with id a2ef35a9e954227445b49998d78567e5aba76885a034f4cc2c45e79fef076d48 Nov 02 19:27:34.413686 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.413551 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b533ad_a1e5_45ef_87ce_a0b2b37b7112.slice/crio-1fcc1c87c926650c79308f45aed6ce979b3e78eeee4bc1763a49139452f6d65d WatchSource:0}: Error finding container 1fcc1c87c926650c79308f45aed6ce979b3e78eeee4bc1763a49139452f6d65d: Status 404 returned error can't find the container with id 1fcc1c87c926650c79308f45aed6ce979b3e78eeee4bc1763a49139452f6d65d Nov 02 19:27:34.413938 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.413914 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9afd80_1122_46ad_8082_77f3b20738e9.slice/crio-555e7d478466bbd651f8f79901a34fd0f6249269c925e5c987d40aaba9fb6729 WatchSource:0}: Error finding container 555e7d478466bbd651f8f79901a34fd0f6249269c925e5c987d40aaba9fb6729: Status 404 returned error can't find the container with id 555e7d478466bbd651f8f79901a34fd0f6249269c925e5c987d40aaba9fb6729 Nov 02 19:27:34.426451 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:27:34.426426 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ef8e16_cb3d_4b88_9a4e_6970f544b650.slice/crio-1e4b6773962956f2d1372da9377a42524ec23f86a2bbe93dd6d5f66a74819134 WatchSource:0}: Error finding container 1e4b6773962956f2d1372da9377a42524ec23f86a2bbe93dd6d5f66a74819134: Status 404 returned error can't find the container with id 1e4b6773962956f2d1372da9377a42524ec23f86a2bbe93dd6d5f66a74819134 Nov 02 19:27:34.741846 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.741758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:34.741846 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.741828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:34.742063 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.741936 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:34.742063 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.741959 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:34.742063 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.742009 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:27:35.741989428 +0000 UTC m=+4.237367941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:34.742063 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.742047 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret podName:fea7ce1f-d9c4-40b4-a160-0f7788c416a6 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:35.742030724 +0000 UTC m=+4.237409238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret") pod "global-pull-secret-syncer-2dw9d" (UID: "fea7ce1f-d9c4-40b4-a160-0f7788c416a6") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:34.842724 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.842687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:34.842887 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.842820 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:34.842887 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.842835 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:34.842887 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.842844 2578 projected.go:196] Error preparing data for projected volume kube-api-access-z8qhm for pod openshift-network-diagnostics/network-check-target-wmg7k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:34.842995 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:34.842894 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm podName:1fdecbd0-dcd5-42f0-85ce-57b32e400992 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:35.842881013 +0000 UTC m=+4.338259526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8qhm" (UniqueName: "kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm") pod "network-check-target-wmg7k" (UID: "1fdecbd0-dcd5-42f0-85ce-57b32e400992") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:34.998393 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:34.998308 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rglh2" event={"ID":"3fc1dba1-1e3a-4639-b2da-8277acf64efb","Type":"ContainerStarted","Data":"bec747754e25bc4ba2d55d70e7de3ed5b90896396a5fcc7a48e1308c9e44a591"} Nov 02 19:27:35.000090 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.000060 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j585w" event={"ID":"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112","Type":"ContainerStarted","Data":"1fcc1c87c926650c79308f45aed6ce979b3e78eeee4bc1763a49139452f6d65d"} Nov 02 19:27:35.001446 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.001416 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"a2ef35a9e954227445b49998d78567e5aba76885a034f4cc2c45e79fef076d48"} Nov 02 19:27:35.002800 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.002767 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfpnh" event={"ID":"d14e549d-addd-40f6-b6af-12582e1a028b","Type":"ContainerStarted","Data":"4e2b0c34247ff094e999c42aedf01d2c017c032ed54df4e40ecc6e722f59cca7"} Nov 02 19:27:35.004026 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.004004 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4b6j" event={"ID":"52b4ca7a-3abb-4c17-a460-51f0b45cadca","Type":"ContainerStarted","Data":"fccae350b6edf37351f93eaa658b8873d4e6f1a8e850b6a0b66ae8b19472e4d2"} Nov 02 19:27:35.005364 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.005320 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" event={"ID":"82ef8e16-cb3d-4b88-9a4e-6970f544b650","Type":"ContainerStarted","Data":"1e4b6773962956f2d1372da9377a42524ec23f86a2bbe93dd6d5f66a74819134"} Nov 02 19:27:35.006795 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.006769 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerStarted","Data":"555e7d478466bbd651f8f79901a34fd0f6249269c925e5c987d40aaba9fb6729"} Nov 02 19:27:35.008358 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.008333 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdlx5" event={"ID":"c4a482a0-025f-4bd6-8229-aa6971510e12","Type":"ContainerStarted","Data":"671c112c9149b2bd8db675a1c244110b2d1173ed435e64b6a8ba1b33ac5d2ac7"} Nov 02 19:27:35.009829 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.009801 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-789jj" event={"ID":"9e7785c4-f87a-4969-9d54-1aafb3c2e641","Type":"ContainerStarted","Data":"50c6edd275cac77e2abb2a93709a893bdb350a36d624d0cc3f23589c6cab21fc"} Nov 02 19:27:35.142315 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.142240 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:33 +0000 UTC" deadline="2028-08-03 06:20:40.682929442 +0000 UTC" Nov 02 19:27:35.142315 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.142313 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15346h53m5.540620134s" Nov 02 19:27:35.750956 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.750921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:35.751066 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.750988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:35.751143 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.751128 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:35.751202 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.751189 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:27:37.751169829 +0000 UTC m=+6.246548333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:35.751276 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.751263 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:35.751347 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.751322 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret podName:fea7ce1f-d9c4-40b4-a160-0f7788c416a6 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:37.751309744 +0000 UTC m=+6.246688263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret") pod "global-pull-secret-syncer-2dw9d" (UID: "fea7ce1f-d9c4-40b4-a160-0f7788c416a6") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:35.851952 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.851916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:35.852157 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.852142 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:35.852208 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.852167 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:35.852208 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.852181 2578 projected.go:196] Error preparing data for projected volume kube-api-access-z8qhm for pod openshift-network-diagnostics/network-check-target-wmg7k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:35.852271 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.852242 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm podName:1fdecbd0-dcd5-42f0-85ce-57b32e400992 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:37.852220564 +0000 UTC m=+6.347599087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8qhm" (UniqueName: "kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm") pod "network-check-target-wmg7k" (UID: "1fdecbd0-dcd5-42f0-85ce-57b32e400992") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:35.991406 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.991341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:35.991535 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.991449 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:35.991571 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.991557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:35.991625 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.991609 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:35.991681 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:35.991670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:35.991735 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:35.991723 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:36.018866 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:36.018094 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"bbbf4706e41de4207b1a770ebd3e743487e0db0d482f4edb6b9f8bcdd0ad3388"} Nov 02 19:27:36.142912 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:36.142842 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:33 +0000 UTC" deadline="2028-08-02 01:47:32.821731032 +0000 UTC" Nov 02 19:27:36.142912 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:36.142903 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15318h19m56.67883256s" Nov 02 19:27:37.034153 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.034117 2578 generic.go:356] "Generic (PLEG): container finished" podID="747e2de570be6f488466605f00369eb2" containerID="2b82d5eedd150e3e307eb7c46fc9c42269eb4a0132c4a141b7f9a8e4ad6a930c" exitCode=0 Nov 02 19:27:37.035132 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.035089 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerDied","Data":"2b82d5eedd150e3e307eb7c46fc9c42269eb4a0132c4a141b7f9a8e4ad6a930c"} Nov 02 19:27:37.149871 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.149749 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" podStartSLOduration=4.149728791 podStartE2EDuration="4.149728791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:36.133429144 +0000 UTC m=+4.628807667" watchObservedRunningTime="2026-11-02 19:27:37.149728791 +0000 UTC m=+5.645107305" Nov 02 19:27:37.772811 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.772778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:37.772977 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.772842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:37.772977 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.772948 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:37.773077 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.773018 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret podName:fea7ce1f-d9c4-40b4-a160-0f7788c416a6 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:41.772998605 +0000 UTC m=+10.268377133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret") pod "global-pull-secret-syncer-2dw9d" (UID: "fea7ce1f-d9c4-40b4-a160-0f7788c416a6") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:37.773077 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.773018 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:37.773077 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.773072 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:27:41.773057153 +0000 UTC m=+10.268435658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:37.873202 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.873173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:37.873371 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.873316 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:37.873371 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.873333 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:37.873371 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.873343 2578 projected.go:196] Error preparing data for projected volume kube-api-access-z8qhm for pod openshift-network-diagnostics/network-check-target-wmg7k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:37.873529 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.873381 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm podName:1fdecbd0-dcd5-42f0-85ce-57b32e400992 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:41.873368912 +0000 UTC m=+10.368747413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8qhm" (UniqueName: "kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm") pod "network-check-target-wmg7k" (UID: "1fdecbd0-dcd5-42f0-85ce-57b32e400992") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:37.992001 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.989772 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:37.992001 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.989771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:37.992001 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:37.989986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:37.992001 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.989924 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:37.992001 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.990074 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:37.992001 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:37.990248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:38.047707 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:38.047665 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"5b970eab0b7694596f8d763910ac4520ac56f6f0d9cb2d6412802669d3cc0127"} Nov 02 19:27:38.053266 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:38.053236 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerStarted","Data":"71b7414f79c8ff409863a766bac24d23a03f260fc48ce708ec81ae7d3919c1a0"} Nov 02 19:27:38.281694 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:38.281593 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" podStartSLOduration=5.281572368 podStartE2EDuration="5.281572368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:38.16359753 +0000 UTC m=+6.658976069" watchObservedRunningTime="2026-11-02 19:27:38.281572368 +0000 UTC m=+6.776950913" Nov 02 19:27:39.062397 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:39.062359 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="71b7414f79c8ff409863a766bac24d23a03f260fc48ce708ec81ae7d3919c1a0" exitCode=0 Nov 02 19:27:39.063531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:39.063502 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerDied","Data":"71b7414f79c8ff409863a766bac24d23a03f260fc48ce708ec81ae7d3919c1a0"} Nov 02 19:27:39.991958 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:39.991462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:39.991958 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:39.991492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:39.991958 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:39.991603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:39.991958 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:39.991640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:39.991958 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:39.991712 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:39.991958 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:39.991792 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:40.076351 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:40.076310 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xfpnh" event={"ID":"d14e549d-addd-40f6-b6af-12582e1a028b","Type":"ContainerStarted","Data":"f95be03753773be24d5a59d6c6100753e2f0615244dea25c1b329c6be60d2df4"} Nov 02 19:27:41.089248 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.089165 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j585w" event={"ID":"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112","Type":"ContainerStarted","Data":"45eb162d3bfdbf79e22e1acbd03f432dad9f381349b6dff16f85272c69c0e57f"} Nov 02 19:27:41.093038 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.093009 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" event={"ID":"82ef8e16-cb3d-4b88-9a4e-6970f544b650","Type":"ContainerStarted","Data":"ca8b76f8e6649f382d29e8fffdc829d5de4af4d22bbae3126f7b6079b3a04ff3"} Nov 02 19:27:41.202911 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.202850 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xfpnh" podStartSLOduration=4.235332713 podStartE2EDuration="9.202831095s" totalImagesPullingTime="4.967498382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:40.193798584 +0000 UTC m=+8.689177107" watchObservedRunningTime="2026-11-02 19:27:41.202831095 +0000 UTC m=+9.698209620" Nov 02 19:27:41.203519 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.203476 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j585w" podStartSLOduration=3.710737827 podStartE2EDuration="9.203460946s" totalImagesPullingTime="5.492723119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:41.202414348 +0000 UTC m=+9.697792871" watchObservedRunningTime="2026-11-02 19:27:41.203460946 +0000 UTC m=+9.698839470" Nov 02 19:27:41.807423 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.807382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:41.807603 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.807446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:41.807663 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.807606 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:41.807715 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.807665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:27:49.807647109 +0000 UTC m=+18.303025625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:41.808053 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.808035 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:41.808164 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.808082 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret podName:fea7ce1f-d9c4-40b4-a160-0f7788c416a6 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:49.808067402 +0000 UTC m=+18.303445907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret") pod "global-pull-secret-syncer-2dw9d" (UID: "fea7ce1f-d9c4-40b4-a160-0f7788c416a6") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:41.908159 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.908119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:41.908342 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.908312 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:41.908342 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.908333 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:41.908512 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.908346 2578 projected.go:196] Error preparing data for projected volume kube-api-access-z8qhm for pod openshift-network-diagnostics/network-check-target-wmg7k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:41.908512 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.908407 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm podName:1fdecbd0-dcd5-42f0-85ce-57b32e400992 nodeName:}" failed. No retries permitted until 2026-11-02 19:27:49.908387984 +0000 UTC m=+18.403766498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8qhm" (UniqueName: "kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm") pod "network-check-target-wmg7k" (UID: "1fdecbd0-dcd5-42f0-85ce-57b32e400992") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:41.998698 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.998668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:41.999017 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.998988 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:41.999495 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.999476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:41.999700 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:41.999523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:41.999866 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:41.999847 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:42.000241 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:42.000182 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:42.727727 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:42.727699 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:27:42.979154 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:42.978834 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:27:42.727720007Z","UUID":"7cbcc7e5-fbc6-433f-ba4a-6131e17c2f46","Handler":null,"Name":"","Endpoint":""} Nov 02 19:27:42.980858 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:42.980512 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:27:42.980858 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:42.980544 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:27:43.099996 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.099913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4b6j" event={"ID":"52b4ca7a-3abb-4c17-a460-51f0b45cadca","Type":"ContainerStarted","Data":"632498539a8d977a35cd3cd90c338ee1bdc0786d63f07d9b690dc10b8791edba"} Nov 02 19:27:43.102317 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.102276 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" event={"ID":"82ef8e16-cb3d-4b88-9a4e-6970f544b650","Type":"ContainerStarted","Data":"f57d7cdad9c5f3e93146f2a2590834bcc7beb36abd0ec1a8f874c02eaa164978"} Nov 02 19:27:43.104219 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.104196 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-789jj" event={"ID":"9e7785c4-f87a-4969-9d54-1aafb3c2e641","Type":"ContainerStarted","Data":"b20c91d3ea3e26b92bf16aab1c18c76007f13c268f03c310073d228724508cf5"} Nov 02 19:27:43.213704 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.213644 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s4b6j" podStartSLOduration=3.026084745 podStartE2EDuration="11.213627654s" totalImagesPullingTime="8.187542909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:43.21265631 +0000 UTC m=+11.708034832" watchObservedRunningTime="2026-11-02 19:27:43.213627654 +0000 UTC m=+11.709006176" Nov 02 19:27:43.330686 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.330628 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-789jj" podStartSLOduration=3.175170088 podStartE2EDuration="11.330610154s" totalImagesPullingTime="8.155440066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:43.330505282 +0000 UTC m=+11.825883804" watchObservedRunningTime="2026-11-02 19:27:43.330610154 +0000 UTC m=+11.825988680" Nov 02 19:27:43.649982 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.649916 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:43.650487 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.650464 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:43.989417 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.989383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:43.989417 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.989419 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:43.989943 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:43.989382 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:43.989943 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:43.989535 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:43.989943 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:43.989603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:43.989943 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:43.989739 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:44.109545 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:44.109510 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="93f99487e1b0f52cff574e420a9c5931bb98a87638e478a8a810fca58a125072" exitCode=0 Nov 02 19:27:44.111180 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:44.109855 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerDied","Data":"93f99487e1b0f52cff574e420a9c5931bb98a87638e478a8a810fca58a125072"} Nov 02 19:27:44.111180 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:44.110939 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:44.111604 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:44.111582 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:27:45.113635 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.113603 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" event={"ID":"82ef8e16-cb3d-4b88-9a4e-6970f544b650","Type":"ContainerStarted","Data":"0b5369be98f46b0fb436c52ad74db44c2c13531faacddb1a6c024d3649444be3"} Nov 02 19:27:45.115076 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.115051 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdlx5" event={"ID":"c4a482a0-025f-4bd6-8229-aa6971510e12","Type":"ContainerStarted","Data":"9e7ea32a08261da7c6d113e598fd579cd6b8fec7ccbafa719ff50d26bf1ded7d"} Nov 02 19:27:45.116453 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.116432 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rglh2" event={"ID":"3fc1dba1-1e3a-4639-b2da-8277acf64efb","Type":"ContainerStarted","Data":"806fc1bfa34570d76502a16de8109563acd6781baf7e1f53ff9acb0f6013d451"} Nov 02 19:27:45.229599 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.229543 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" podStartSLOduration=3.733978525 podStartE2EDuration="13.229524451s" totalImagesPullingTime="9.495545926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:45.228948626 +0000 UTC m=+13.724327148" watchObservedRunningTime="2026-11-02 19:27:45.229524451 +0000 UTC m=+13.724902974" Nov 02 19:27:45.342824 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.342737 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jdlx5" podStartSLOduration=4.904343599 podStartE2EDuration="13.342705431s" totalImagesPullingTime="8.438361832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:45.342474615 +0000 UTC m=+13.837853137" watchObservedRunningTime="2026-11-02 19:27:45.342705431 +0000 UTC m=+13.838083953" Nov 02 19:27:45.459396 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.459335 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rglh2" podStartSLOduration=2.890578042 podStartE2EDuration="13.459315707s" totalImagesPullingTime="10.568737665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:45.458908194 +0000 UTC m=+13.954286716" watchObservedRunningTime="2026-11-02 19:27:45.459315707 +0000 UTC m=+13.954694231" Nov 02 19:27:45.989265 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.989182 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:45.989265 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.989204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:45.989265 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:45.989240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:45.989539 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:45.989355 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:45.989734 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:45.989703 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:45.989862 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:45.989820 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:46.123777 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:46.123734 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="94f9fa9fb8831fee0e0ef24e0264bbab413a8ede5ba0aeec5738005aa1d6f450" exitCode=0 Nov 02 19:27:46.124327 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:46.124284 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerDied","Data":"94f9fa9fb8831fee0e0ef24e0264bbab413a8ede5ba0aeec5738005aa1d6f450"} Nov 02 19:27:47.988973 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:47.988765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:47.989442 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:47.988765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:47.989442 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:47.989089 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:47.989442 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:47.988811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:47.989442 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:47.989166 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:47.989442 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:47.989252 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:48.129970 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:48.129936 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="c2306260e4aef8ae6f6ef0e4650e4c479e08447cd110361485f9ed647b964c79" exitCode=0 Nov 02 19:27:48.130122 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:48.129983 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerDied","Data":"c2306260e4aef8ae6f6ef0e4650e4c479e08447cd110361485f9ed647b964c79"} Nov 02 19:27:49.870706 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:49.870469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:49.870706 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:49.870693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:49.871193 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.870611 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:49.871193 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.870797 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:49.871193 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.870810 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret podName:fea7ce1f-d9c4-40b4-a160-0f7788c416a6 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:05.870792094 +0000 UTC m=+34.366170608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret") pod "global-pull-secret-syncer-2dw9d" (UID: "fea7ce1f-d9c4-40b4-a160-0f7788c416a6") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:27:49.871193 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.870838 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:28:05.870825447 +0000 UTC m=+34.366203947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:27:49.972008 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:49.971971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:49.972184 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.972129 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:27:49.972184 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.972152 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:27:49.972184 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.972163 2578 projected.go:196] Error preparing data for projected volume kube-api-access-z8qhm for pod openshift-network-diagnostics/network-check-target-wmg7k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:49.972348 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.972215 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm podName:1fdecbd0-dcd5-42f0-85ce-57b32e400992 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:05.97219908 +0000 UTC m=+34.467577584 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8qhm" (UniqueName: "kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm") pod "network-check-target-wmg7k" (UID: "1fdecbd0-dcd5-42f0-85ce-57b32e400992") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:27:49.989259 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:49.989231 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:49.989405 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:49.989267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:49.989405 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.989383 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:49.989519 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.989490 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:49.989602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:49.989539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:49.989674 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:49.989645 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:50.137891 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:50.137795 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"99a6bd036adefbcb8c85a74e32fad15f83b4c20231a01f97ab097a04cbf604ee"} Nov 02 19:27:50.138044 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:50.137895 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"08af4367c14b621d70dfbf20aeb14b9f4bb1baf7037ddfe8c8819beb96ff0db7"} Nov 02 19:27:50.138044 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:50.137940 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"3c563e02f89fcf471ecd7af74a6a861016aa4e102d2716b17163e44afad4755c"} Nov 02 19:27:50.138044 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:50.137982 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"c9caf79124e575d590e459da6dd7325abd28eb5fce64015a814981c2118ca703"} Nov 02 19:27:50.138044 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:50.138024 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"57960b539af745ba589851ae90d2d96a54c9b9e70b59ff8ad46e92f8a0500963"} Nov 02 19:27:50.138222 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:50.138059 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"459252d7dde35ce4a492890a4d2bc2b8952a2d69dd05fc1e323b3e98ec4c8db9"} Nov 02 19:27:51.142649 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:51.142619 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="a7878bd202102eb54b55adf4906d5df79012f3ab389038484076fa36277915f5" exitCode=0 Nov 02 19:27:51.143175 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:51.142658 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerDied","Data":"a7878bd202102eb54b55adf4906d5df79012f3ab389038484076fa36277915f5"} Nov 02 19:27:51.989490 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:51.989445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:51.989642 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:51.989545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:51.989642 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:51.989561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:51.989642 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:51.989585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:51.989766 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:51.989660 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:51.989766 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:51.989727 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:52.148227 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:52.148193 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"233a6f9f4b63d310fce93f69e09b296283a3fbf0a22797347bc2854c07af979d"} Nov 02 19:27:52.151120 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:52.151090 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="82c606c91fc7df34aba7d0b47483596abdafdcbb83949eba08375a05766cc2df" exitCode=0 Nov 02 19:27:52.151258 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:52.151130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerDied","Data":"82c606c91fc7df34aba7d0b47483596abdafdcbb83949eba08375a05766cc2df"} Nov 02 19:27:53.156674 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:53.156640 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zs222" event={"ID":"9a9afd80-1122-46ad-8082-77f3b20738e9","Type":"ContainerStarted","Data":"f574a0bfedf87b3f0db42e4df4b5cd4db216681fca07d682e1c11cc6fdf408d9"} Nov 02 19:27:53.276574 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:53.276523 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zs222" podStartSLOduration=8.156248755 podStartE2EDuration="21.276509198s" totalImagesPullingTime="13.120260443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:53.275930714 +0000 UTC m=+21.771309236" watchObservedRunningTime="2026-11-02 19:27:53.276509198 +0000 UTC m=+21.771887720" Nov 02 19:27:53.989760 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:53.989724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:53.989891 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:53.989765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:53.989891 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:53.989724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:53.989891 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:53.989840 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:53.990001 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:53.989921 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:53.990050 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:53.990033 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:54.162095 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:54.161974 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" event={"ID":"9e81006a-11bf-44aa-8561-1de0236016c8","Type":"ContainerStarted","Data":"b9a605a6fa3ec5de838c11da60cb8ef234d82577158f6edc47a23d3673e5a535"} Nov 02 19:27:54.424988 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:54.424874 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" podStartSLOduration=7.527112228 podStartE2EDuration="22.424860095s" totalImagesPullingTime="14.897747867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:27:54.374078544 +0000 UTC m=+22.869457066" watchObservedRunningTime="2026-11-02 19:27:54.424860095 +0000 UTC m=+22.920238617" Nov 02 19:27:55.163777 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.163753 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:27:55.164160 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.164142 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:55.164235 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.164223 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:55.179359 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.179334 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:55.179481 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.179438 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:27:55.989746 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.989706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:55.989746 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.989722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:55.990030 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:55.989821 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:55.990030 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:55.989869 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:55.990030 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:55.989936 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:55.990156 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:55.990050 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:56.165277 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:56.165254 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:27:56.414378 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:56.414276 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7hnrd"] Nov 02 19:27:56.414533 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:56.414495 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:56.414615 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:56.414600 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:56.416894 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:56.416870 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmg7k"] Nov 02 19:27:56.417009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:56.416960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:56.417044 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:56.417032 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:56.417310 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:56.417280 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2dw9d"] Nov 02 19:27:56.417383 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:56.417372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:56.417465 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:56.417450 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:57.167411 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:57.167382 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:27:57.991534 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:57.991508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:57.991661 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:57.991508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:57.991661 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:57.991599 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:57.991750 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:57.991508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:57.991750 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:57.991673 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:57.991812 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:57.991760 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:27:59.989087 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:59.989056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:27:59.989674 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:59.989056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:27:59.989674 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:59.989284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:27:59.989674 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:27:59.989057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:27:59.989674 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:59.989184 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:27:59.989674 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:27:59.989446 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:28:01.990310 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:01.990132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:01.990755 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:01.990220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:01.990755 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:01.990387 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wmg7k" podUID="1fdecbd0-dcd5-42f0-85ce-57b32e400992" Nov 02 19:28:01.990755 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:01.990270 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:01.990755 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:01.990493 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2dw9d" podUID="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" Nov 02 19:28:01.990755 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:01.990609 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7hnrd" podUID="3a889062-1b3c-442a-860d-64e9209a6acf" Nov 02 19:28:02.224249 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.224222 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeReady" Nov 02 19:28:02.224440 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.224360 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 19:28:02.336854 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.336822 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8shl"] Nov 02 19:28:02.340343 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.340327 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8shl" Nov 02 19:28:02.387135 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.387074 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-52pkd"] Nov 02 19:28:02.391088 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.391067 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txxgt"] Nov 02 19:28:02.391203 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.391181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-52pkd" Nov 02 19:28:02.394794 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.394775 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86c87c649b-jwznp"] Nov 02 19:28:02.394905 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.394880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-txxgt" Nov 02 19:28:02.398515 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.398494 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p"] Nov 02 19:28:02.400493 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.400464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.402412 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.402393 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-w8zlt"] Nov 02 19:28:02.402521 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.402431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.407026 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.407009 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-9cf6bf74f-dljwv"] Nov 02 19:28:02.407140 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.407118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.410325 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.410291 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g"] Nov 02 19:28:02.410418 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.410387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.413580 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.413562 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs"] Nov 02 19:28:02.413692 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.413675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" Nov 02 19:28:02.416581 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.416564 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn"] Nov 02 19:28:02.416702 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.416688 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.420593 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.420573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.437492 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.437468 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.437611 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.437469 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.438040 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.438019 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n"] Nov 02 19:28:02.441819 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.441799 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jdcml"] Nov 02 19:28:02.441987 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.441971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.445594 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.445570 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-fvrzl"] Nov 02 19:28:02.446575 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.446544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.450397 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.450377 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75f86d58f7-2pbzr"] Nov 02 19:28:02.450573 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.450556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:02.453534 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.453503 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7"] Nov 02 19:28:02.453534 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.453513 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.455889 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.455759 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ktd87\"" Nov 02 19:28:02.455889 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.455764 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 19:28:02.455889 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.455863 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 19:28:02.456715 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.456698 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vpjl9"] Nov 02 19:28:02.456800 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.456766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" Nov 02 19:28:02.459897 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.459881 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xmrs7"] Nov 02 19:28:02.459991 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.459977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463678 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-52pkd"] Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463725 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8shl"] Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463748 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txxgt"] Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463774 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-w8zlt"] Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463795 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86c87c649b-jwznp"] Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463822 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p"] Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463842 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn"] Nov 02 19:28:02.465513 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.463957 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.471070 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.470996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh5ms\" (UniqueName: \"kubernetes.io/projected/84e7c05f-e63a-4829-ab28-c25bf61f9193-kube-api-access-rh5ms\") pod \"kas-connection-checker-685895c7c9-k8shl\" (UID: \"84e7c05f-e63a-4829-ab28-c25bf61f9193\") " pod="kube-system/kas-connection-checker-685895c7c9-k8shl" Nov 02 19:28:02.488698 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.488663 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-fvrzl"] Nov 02 19:28:02.493907 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.493597 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jdcml"] Nov 02 19:28:02.495202 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.495176 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7"] Nov 02 19:28:02.495931 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.495908 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmrs7"] Nov 02 19:28:02.496591 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.496568 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g"] Nov 02 19:28:02.497911 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.497703 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f86d58f7-2pbzr"] Nov 02 19:28:02.498392 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.498369 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n"] Nov 02 19:28:02.499884 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.499863 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.499990 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.499864 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.499990 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.499873 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 19:28:02.499990 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.499939 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.499990 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.499975 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.500216 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500024 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.500216 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500031 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 19:28:02.500216 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500050 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.500216 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500069 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.500531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500235 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 19:28:02.500531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500235 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.500531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500260 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.500531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500273 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 19:28:02.500531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500385 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.500531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500481 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.500531 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500532 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.501046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500701 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.501046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500762 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 19:28:02.501046 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.500769 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.550234 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.550200 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 19:28:02.551443 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.551420 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 19:28:02.551568 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.551492 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 19:28:02.552409 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.552391 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs"] Nov 02 19:28:02.553032 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.553018 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9cf6bf74f-dljwv"] Nov 02 19:28:02.553665 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.553640 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpjl9"] Nov 02 19:28:02.554028 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554004 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 19:28:02.554121 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554013 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 19:28:02.554121 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554046 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 19:28:02.554121 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554014 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 19:28:02.554121 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554118 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 19:28:02.554319 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554178 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:28:02.554319 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554205 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fzxnx\"" Nov 02 19:28:02.554319 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.554264 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2skqm\"" Nov 02 19:28:02.556677 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556662 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 19:28:02.556749 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556687 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 19:28:02.556749 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556721 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 19:28:02.556749 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556735 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dc6gd\"" Nov 02 19:28:02.556749 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556745 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:28:02.556885 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556754 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 19:28:02.556885 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556751 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 19:28:02.556885 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.556790 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 19:28:02.572176 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8jm\" (UniqueName: \"kubernetes.io/projected/596ee397-e505-475e-aa1f-6d7096d8312b-kube-api-access-ck8jm\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.572234 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947f945b-c140-4889-aaae-d332185913c7-tmp-dir\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.572234 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.572234 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.572342 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6q8h\" (UniqueName: \"kubernetes.io/projected/2939006c-2a64-4684-968e-dc69c92c23ea-kube-api-access-z6q8h\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.572375 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh5ms\" (UniqueName: \"kubernetes.io/projected/84e7c05f-e63a-4829-ab28-c25bf61f9193-kube-api-access-rh5ms\") pod \"kas-connection-checker-685895c7c9-k8shl\" (UID: \"84e7c05f-e63a-4829-ab28-c25bf61f9193\") " pod="kube-system/kas-connection-checker-685895c7c9-k8shl" Nov 02 19:28:02.572375 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.572433 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.572433 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bf9d232-57b4-42b2-9919-b03b3170fe02-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:02.572433 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gvn\" (UniqueName: \"kubernetes.io/projected/add7fff3-f01d-4f58-893f-7cf024ae59e2-kube-api-access-n4gvn\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.572532 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.572532 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnfpf\" (UniqueName: \"kubernetes.io/projected/1804f9cc-5858-4cbb-a623-af98ee3234c2-kube-api-access-cnfpf\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.572623 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.572623 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.572623 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1804f9cc-5858-4cbb-a623-af98ee3234c2-serving-cert\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.572741 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/add7fff3-f01d-4f58-893f-7cf024ae59e2-config\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.572741 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kkk\" (UniqueName: \"kubernetes.io/projected/189ded48-fc9b-4109-828d-3c3cae8b5242-kube-api-access-s2kkk\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:02.572741 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.572888 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czjl\" (UniqueName: \"kubernetes.io/projected/be44fdaf-13bc-4704-8c58-bcc0359528e8-kube-api-access-6czjl\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.572888 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/add7fff3-f01d-4f58-893f-7cf024ae59e2-serving-cert\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.572888 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.572888 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/947f945b-c140-4889-aaae-d332185913c7-config-volume\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.572888 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.573093 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.573093 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2939006c-2a64-4684-968e-dc69c92c23ea-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.573093 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.573093 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.572991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.573093 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9zfz\" (UniqueName: \"kubernetes.io/projected/947f945b-c140-4889-aaae-d332185913c7-kube-api-access-z9zfz\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.573093 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573088 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189ded48-fc9b-4109-828d-3c3cae8b5242-cert\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:02.573317 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.573317 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.573317 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.573317 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.573317 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.573317 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9cq\" (UniqueName: \"kubernetes.io/projected/b5c10760-26db-48ed-8fe8-92700a8fcf06-kube-api-access-tr9cq\") pod \"volume-data-source-validator-58c9f5f46d-snj5g\" (UID: \"b5c10760-26db-48ed-8fe8-92700a8fcf06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" Nov 02 19:28:02.573482 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-tmp\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.573482 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be54cb41-84fc-4d14-a8c8-0435cf0954c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.573482 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.573482 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-stats-auth\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.573482 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.573482 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjqh\" (UniqueName: \"kubernetes.io/projected/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b-kube-api-access-4rjqh\") pod \"network-check-source-6f78d59f99-mz8m7\" (UID: \"cb731f5a-7ec8-4440-b4b9-2ada95e8d40b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" Nov 02 19:28:02.573645 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzkt\" (UniqueName: \"kubernetes.io/projected/f16e19f1-3da4-43db-b861-7146aa9dbede-kube-api-access-fbzkt\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.573645 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:02.573645 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-config\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.573645 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f16e19f1-3da4-43db-b861-7146aa9dbede-serving-cert\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.573645 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mql6m\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.573645 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45klt\" (UniqueName: \"kubernetes.io/projected/85cc6f7e-1889-40bb-a184-2138db4d29ba-kube-api-access-45klt\") pod \"kas-connection-checker-685895c7c9-52pkd\" (UID: \"85cc6f7e-1889-40bb-a184-2138db4d29ba\") " pod="kube-system/kas-connection-checker-685895c7c9-52pkd" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-default-certificate\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be54cb41-84fc-4d14-a8c8-0435cf0954c9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfpv\" (UniqueName: \"kubernetes.io/projected/be54cb41-84fc-4d14-a8c8-0435cf0954c9-kube-api-access-xgfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-snapshots\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-service-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.573906 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4cqc\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.574178 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.573944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8qp\" (UniqueName: \"kubernetes.io/projected/a3ff325f-d384-407e-af38-2c876d9b686a-kube-api-access-gl8qp\") pod \"kas-connection-checker-685895c7c9-txxgt\" (UID: \"a3ff325f-d384-407e-af38-2c876d9b686a\") " pod="kube-system/kas-connection-checker-685895c7c9-txxgt" Nov 02 19:28:02.599920 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.599892 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.599920 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.599911 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.600069 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.599933 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.600069 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.599953 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.600069 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.599972 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:28:02.600069 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.599991 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:28:02.600069 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.600015 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 19:28:02.600069 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.600034 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 19:28:02.600285 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.600071 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:02.601228 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.601213 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 19:28:02.630350 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.630317 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh5ms\" (UniqueName: \"kubernetes.io/projected/84e7c05f-e63a-4829-ab28-c25bf61f9193-kube-api-access-rh5ms\") pod \"kas-connection-checker-685895c7c9-k8shl\" (UID: \"84e7c05f-e63a-4829-ab28-c25bf61f9193\") " pod="kube-system/kas-connection-checker-685895c7c9-k8shl" Nov 02 19:28:02.653126 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.653064 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k8shl" Nov 02 19:28:02.674293 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzkt\" (UniqueName: \"kubernetes.io/projected/f16e19f1-3da4-43db-b861-7146aa9dbede-kube-api-access-fbzkt\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.674398 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:02.674443 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.674402 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:02.674479 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.674447 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.174431643 +0000 UTC m=+31.669810156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:02.674518 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-config\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.674550 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f16e19f1-3da4-43db-b861-7146aa9dbede-serving-cert\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.674550 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mql6m\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.674605 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.674637 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45klt\" (UniqueName: \"kubernetes.io/projected/85cc6f7e-1889-40bb-a184-2138db4d29ba-kube-api-access-45klt\") pod \"kas-connection-checker-685895c7c9-52pkd\" (UID: \"85cc6f7e-1889-40bb-a184-2138db4d29ba\") " pod="kube-system/kas-connection-checker-685895c7c9-52pkd" Nov 02 19:28:02.674674 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.674724 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-default-certificate\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.674724 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be54cb41-84fc-4d14-a8c8-0435cf0954c9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.674814 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfpv\" (UniqueName: \"kubernetes.io/projected/be54cb41-84fc-4d14-a8c8-0435cf0954c9-kube-api-access-xgfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.674814 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-snapshots\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.674814 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-service-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.674948 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.674814 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.174793872 +0000 UTC m=+31.670172431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:02.674948 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4cqc\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.674948 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8qp\" (UniqueName: \"kubernetes.io/projected/a3ff325f-d384-407e-af38-2c876d9b686a-kube-api-access-gl8qp\") pod \"kas-connection-checker-685895c7c9-txxgt\" (UID: \"a3ff325f-d384-407e-af38-2c876d9b686a\") " pod="kube-system/kas-connection-checker-685895c7c9-txxgt" Nov 02 19:28:02.674948 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8jm\" (UniqueName: \"kubernetes.io/projected/596ee397-e505-475e-aa1f-6d7096d8312b-kube-api-access-ck8jm\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.674948 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947f945b-c140-4889-aaae-d332185913c7-tmp-dir\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.675147 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.675147 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.674992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.675147 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q8h\" (UniqueName: \"kubernetes.io/projected/2939006c-2a64-4684-968e-dc69c92c23ea-kube-api-access-z6q8h\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.675147 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.675357 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.675421 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bf9d232-57b4-42b2-9919-b03b3170fe02-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:02.675421 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gvn\" (UniqueName: \"kubernetes.io/projected/add7fff3-f01d-4f58-893f-7cf024ae59e2-kube-api-access-n4gvn\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.675421 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.675539 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfpf\" (UniqueName: \"kubernetes.io/projected/1804f9cc-5858-4cbb-a623-af98ee3234c2-kube-api-access-cnfpf\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.675539 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.675539 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.675539 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.675508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1804f9cc-5858-4cbb-a623-af98ee3234c2-serving-cert\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.676219 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.676076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-snapshots\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.676219 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.676096 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-config\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.676219 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.676126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-service-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.676453 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.676283 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be54cb41-84fc-4d14-a8c8-0435cf0954c9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.676453 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.676396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947f945b-c140-4889-aaae-d332185913c7-tmp-dir\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.676566 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.676486 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:02.676566 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.676541 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.176526065 +0000 UTC m=+31.671904565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:02.676830 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.676809 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:02.676902 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.676833 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:02.676902 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.676875 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.176861689 +0000 UTC m=+31.672240220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:02.677011 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.676956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.677440 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/add7fff3-f01d-4f58-893f-7cf024ae59e2-config\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.677515 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kkk\" (UniqueName: \"kubernetes.io/projected/189ded48-fc9b-4109-828d-3c3cae8b5242-kube-api-access-s2kkk\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:02.677515 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-default-certificate\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.677515 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.677684 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czjl\" (UniqueName: \"kubernetes.io/projected/be44fdaf-13bc-4704-8c58-bcc0359528e8-kube-api-access-6czjl\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.677684 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bf9d232-57b4-42b2-9919-b03b3170fe02-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:02.677684 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/add7fff3-f01d-4f58-893f-7cf024ae59e2-serving-cert\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.677684 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.677684 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/947f945b-c140-4889-aaae-d332185913c7-config-volume\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.677684 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.677684 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.678025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2939006c-2a64-4684-968e-dc69c92c23ea-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.678025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.678025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.678025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9zfz\" (UniqueName: \"kubernetes.io/projected/947f945b-c140-4889-aaae-d332185913c7-kube-api-access-z9zfz\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.678025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189ded48-fc9b-4109-828d-3c3cae8b5242-cert\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:02.678025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.677965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.678025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9cq\" (UniqueName: \"kubernetes.io/projected/b5c10760-26db-48ed-8fe8-92700a8fcf06-kube-api-access-tr9cq\") pod \"volume-data-source-validator-58c9f5f46d-snj5g\" (UID: \"b5c10760-26db-48ed-8fe8-92700a8fcf06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-tmp\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be54cb41-84fc-4d14-a8c8-0435cf0954c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-stats-auth\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/add7fff3-f01d-4f58-893f-7cf024ae59e2-config\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678251 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/947f945b-c140-4889-aaae-d332185913c7-config-volume\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjqh\" (UniqueName: \"kubernetes.io/projected/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b-kube-api-access-4rjqh\") pod \"network-check-source-6f78d59f99-mz8m7\" (UID: \"cb731f5a-7ec8-4440-b4b9-2ada95e8d40b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" Nov 02 19:28:02.678390 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.678345 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:02.679025 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.678399 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.178382065 +0000 UTC m=+31.673760648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:02.679025 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.678477 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:02.679025 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.678513 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.17850113 +0000 UTC m=+31.673879689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:02.679025 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.678650 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:02.679025 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.678665 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:02.679025 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.678721 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.178706595 +0000 UTC m=+31.674085114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:02.679025 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.678930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1804f9cc-5858-4cbb-a623-af98ee3234c2-serving-cert\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.679427 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.679192 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:02.679427 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:02.679259 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:03.179245625 +0000 UTC m=+31.674624125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:02.679559 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.679530 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.679813 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.679788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.680330 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.680284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.680449 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.680429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2939006c-2a64-4684-968e-dc69c92c23ea-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.680719 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.680702 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.680752 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.680736 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.681104 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.681081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.681217 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.681201 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/add7fff3-f01d-4f58-893f-7cf024ae59e2-serving-cert\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.681593 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.681565 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.681673 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.681638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.682497 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.682391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f16e19f1-3da4-43db-b861-7146aa9dbede-serving-cert\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.683553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.682889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.683553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.683071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.683553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.683148 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-tmp\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.683728 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.683609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.683940 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.683916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-stats-auth\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.684056 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.684035 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be54cb41-84fc-4d14-a8c8-0435cf0954c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.684503 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.684485 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.685026 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.685011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189ded48-fc9b-4109-828d-3c3cae8b5242-cert\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:02.736748 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.736700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mql6m\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.740227 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.740198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8jm\" (UniqueName: \"kubernetes.io/projected/596ee397-e505-475e-aa1f-6d7096d8312b-kube-api-access-ck8jm\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:02.740775 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.740748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzkt\" (UniqueName: \"kubernetes.io/projected/f16e19f1-3da4-43db-b861-7146aa9dbede-kube-api-access-fbzkt\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:02.740869 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.740811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4cqc\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.741189 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.741165 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjqh\" (UniqueName: \"kubernetes.io/projected/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b-kube-api-access-4rjqh\") pod \"network-check-source-6f78d59f99-mz8m7\" (UID: \"cb731f5a-7ec8-4440-b4b9-2ada95e8d40b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" Nov 02 19:28:02.741265 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.741183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45klt\" (UniqueName: \"kubernetes.io/projected/85cc6f7e-1889-40bb-a184-2138db4d29ba-kube-api-access-45klt\") pod \"kas-connection-checker-685895c7c9-52pkd\" (UID: \"85cc6f7e-1889-40bb-a184-2138db4d29ba\") " pod="kube-system/kas-connection-checker-685895c7c9-52pkd" Nov 02 19:28:02.741828 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.741795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:02.741828 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.741803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9cq\" (UniqueName: \"kubernetes.io/projected/b5c10760-26db-48ed-8fe8-92700a8fcf06-kube-api-access-tr9cq\") pod \"volume-data-source-validator-58c9f5f46d-snj5g\" (UID: \"b5c10760-26db-48ed-8fe8-92700a8fcf06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gvn\" (UniqueName: \"kubernetes.io/projected/add7fff3-f01d-4f58-893f-7cf024ae59e2-kube-api-access-n4gvn\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfpf\" (UniqueName: \"kubernetes.io/projected/1804f9cc-5858-4cbb-a623-af98ee3234c2-kube-api-access-cnfpf\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfpv\" (UniqueName: \"kubernetes.io/projected/be54cb41-84fc-4d14-a8c8-0435cf0954c9-kube-api-access-xgfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742286 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8qp\" (UniqueName: \"kubernetes.io/projected/a3ff325f-d384-407e-af38-2c876d9b686a-kube-api-access-gl8qp\") pod \"kas-connection-checker-685895c7c9-txxgt\" (UID: \"a3ff325f-d384-407e-af38-2c876d9b686a\") " pod="kube-system/kas-connection-checker-685895c7c9-txxgt" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742768 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q8h\" (UniqueName: \"kubernetes.io/projected/2939006c-2a64-4684-968e-dc69c92c23ea-kube-api-access-z6q8h\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czjl\" (UniqueName: \"kubernetes.io/projected/be44fdaf-13bc-4704-8c58-bcc0359528e8-kube-api-access-6czjl\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:02.742950 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.742909 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kkk\" (UniqueName: \"kubernetes.io/projected/189ded48-fc9b-4109-828d-3c3cae8b5242-kube-api-access-s2kkk\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:02.744098 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.744079 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9zfz\" (UniqueName: \"kubernetes.io/projected/947f945b-c140-4889-aaae-d332185913c7-kube-api-access-z9zfz\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:02.746979 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.746964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" Nov 02 19:28:02.756250 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.756232 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:02.770912 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.770895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:02.779250 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.779098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:02.806499 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.806467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" Nov 02 19:28:02.813602 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:02.813573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:03.006217 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.006189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-52pkd" Nov 02 19:28:03.012066 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.012048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-txxgt" Nov 02 19:28:03.031543 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.031515 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:03.034763 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.034739 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k8shl"] Nov 02 19:28:03.035135 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.035105 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e7c05f_e63a_4829_ab28_c25bf61f9193.slice/crio-26cdce1aa97ace90c9ebf3d9eb6f76b419a44958e235830ce0cf766201dbcbfe WatchSource:0}: Error finding container 26cdce1aa97ace90c9ebf3d9eb6f76b419a44958e235830ce0cf766201dbcbfe: Status 404 returned error can't find the container with id 26cdce1aa97ace90c9ebf3d9eb6f76b419a44958e235830ce0cf766201dbcbfe Nov 02 19:28:03.058166 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.058132 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:03.058464 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.058446 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:03.079378 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.079167 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:03.118133 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.116310 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g"] Nov 02 19:28:03.118133 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.117053 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c10760_26db_48ed_8fe8_92700a8fcf06.slice/crio-5ef2d819a1b2eb5b4a1d7f164b35868b83f8e06b0ba5504e6d3d0ebe1eab9457 WatchSource:0}: Error finding container 5ef2d819a1b2eb5b4a1d7f164b35868b83f8e06b0ba5504e6d3d0ebe1eab9457: Status 404 returned error can't find the container with id 5ef2d819a1b2eb5b4a1d7f164b35868b83f8e06b0ba5504e6d3d0ebe1eab9457 Nov 02 19:28:03.126676 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.126654 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs"] Nov 02 19:28:03.169429 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.169395 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd7fff3_f01d_4f58_893f_7cf024ae59e2.slice/crio-ff4d1280f420614c96c12e23f03cdcbace6f9b6215c873e9e623625a3e94f9e3 WatchSource:0}: Error finding container ff4d1280f420614c96c12e23f03cdcbace6f9b6215c873e9e623625a3e94f9e3: Status 404 returned error can't find the container with id ff4d1280f420614c96c12e23f03cdcbace6f9b6215c873e9e623625a3e94f9e3 Nov 02 19:28:03.170056 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.170033 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1804f9cc_5858_4cbb_a623_af98ee3234c2.slice/crio-ff4158ee425ebd108bb44e0dd15d482d3ca5176078dc284c3dbeeeae41083af1 WatchSource:0}: Error finding container ff4158ee425ebd108bb44e0dd15d482d3ca5176078dc284c3dbeeeae41083af1: Status 404 returned error can't find the container with id ff4158ee425ebd108bb44e0dd15d482d3ca5176078dc284c3dbeeeae41083af1 Nov 02 19:28:03.171067 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.171022 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jdcml"] Nov 02 19:28:03.172129 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.172106 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n"] Nov 02 19:28:03.180352 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.180325 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" event={"ID":"add7fff3-f01d-4f58-893f-7cf024ae59e2","Type":"ContainerStarted","Data":"ff4d1280f420614c96c12e23f03cdcbace6f9b6215c873e9e623625a3e94f9e3"} Nov 02 19:28:03.181430 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.181410 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" event={"ID":"1804f9cc-5858-4cbb-a623-af98ee3234c2","Type":"ContainerStarted","Data":"ff4158ee425ebd108bb44e0dd15d482d3ca5176078dc284c3dbeeeae41083af1"} Nov 02 19:28:03.182569 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.182548 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" event={"ID":"be54cb41-84fc-4d14-a8c8-0435cf0954c9","Type":"ContainerStarted","Data":"a4daae0fffba6f20b63e1dc34510996a2be78cf4bc6057ebb318e094c795d4ba"} Nov 02 19:28:03.183221 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:03.183328 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:03.183384 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:03.183445 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:03.183445 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183413 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:03.183445 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:03.183445 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183430 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183480 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.183461266 +0000 UTC m=+32.678839774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183507 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183512 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183523 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183531 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183545 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.183532257 +0000 UTC m=+32.678910772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183570 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.183553976 +0000 UTC m=+32.678932494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:03.183629 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183629 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.183615173 +0000 UTC m=+32.678993686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183680 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183736 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183743 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.183729892 +0000 UTC m=+32.679108400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183766 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.183756658 +0000 UTC m=+32.679135163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183783 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.183774815 +0000 UTC m=+32.679153321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183777 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" event={"ID":"b5c10760-26db-48ed-8fe8-92700a8fcf06","Type":"ContainerStarted","Data":"5ef2d819a1b2eb5b4a1d7f164b35868b83f8e06b0ba5504e6d3d0ebe1eab9457"} Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.183849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183947 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:03.184016 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:03.183984 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:04.1839725 +0000 UTC m=+32.679351014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:03.186199 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.186178 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8shl" event={"ID":"84e7c05f-e63a-4829-ab28-c25bf61f9193","Type":"ContainerStarted","Data":"26cdce1aa97ace90c9ebf3d9eb6f76b419a44958e235830ce0cf766201dbcbfe"} Nov 02 19:28:03.209980 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.209959 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb731f5a_7ec8_4440_b4b9_2ada95e8d40b.slice/crio-03b436b1666a8957c1c82f9d9cdb3a2ae86c2aa3111fa67e71ab07930c229336 WatchSource:0}: Error finding container 03b436b1666a8957c1c82f9d9cdb3a2ae86c2aa3111fa67e71ab07930c229336: Status 404 returned error can't find the container with id 03b436b1666a8957c1c82f9d9cdb3a2ae86c2aa3111fa67e71ab07930c229336 Nov 02 19:28:03.211827 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.211809 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7"] Nov 02 19:28:03.220284 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.220262 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189ded48_fc9b_4109_828d_3c3cae8b5242.slice/crio-87c22c1e5b4990baab7426a46d15134e8cfc92138d38f5235d2a97efc1a0ca07 WatchSource:0}: Error finding container 87c22c1e5b4990baab7426a46d15134e8cfc92138d38f5235d2a97efc1a0ca07: Status 404 returned error can't find the container with id 87c22c1e5b4990baab7426a46d15134e8cfc92138d38f5235d2a97efc1a0ca07 Nov 02 19:28:03.221699 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.221683 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpjl9"] Nov 02 19:28:03.384811 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.384783 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cc6f7e_1889_40bb_a184_2138db4d29ba.slice/crio-228f642268f3ba6617452775130c97b2227f551091f8516a1d2f92cb013d9b2e WatchSource:0}: Error finding container 228f642268f3ba6617452775130c97b2227f551091f8516a1d2f92cb013d9b2e: Status 404 returned error can't find the container with id 228f642268f3ba6617452775130c97b2227f551091f8516a1d2f92cb013d9b2e Nov 02 19:28:03.386845 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.386816 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-52pkd"] Nov 02 19:28:03.393640 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.393587 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ff325f_d384_407e_af38_2c876d9b686a.slice/crio-5e7244810ec14b9b66e58ce0518dce2254ad5a1b9b2f84b8de5a162c29e21f28 WatchSource:0}: Error finding container 5e7244810ec14b9b66e58ce0518dce2254ad5a1b9b2f84b8de5a162c29e21f28: Status 404 returned error can't find the container with id 5e7244810ec14b9b66e58ce0518dce2254ad5a1b9b2f84b8de5a162c29e21f28 Nov 02 19:28:03.395071 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.395043 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-txxgt"] Nov 02 19:28:03.414814 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:03.414745 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16e19f1_3da4_43db_b861_7146aa9dbede.slice/crio-a780a4583a03dc51d5ff9a18f01711d63109e4f1e81debbcb77872106b61c1ff WatchSource:0}: Error finding container a780a4583a03dc51d5ff9a18f01711d63109e4f1e81debbcb77872106b61c1ff: Status 404 returned error can't find the container with id a780a4583a03dc51d5ff9a18f01711d63109e4f1e81debbcb77872106b61c1ff Nov 02 19:28:03.414967 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.414943 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-w8zlt"] Nov 02 19:28:03.989367 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.989336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:03.989561 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.989345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:03.989561 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:03.989513 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:04.042214 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.042179 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbvgr\"" Nov 02 19:28:04.042214 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.042186 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:28:04.042625 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.042287 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:28:04.042625 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.042330 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n2mjf\"" Nov 02 19:28:04.042625 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.042377 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c8nwr\"" Nov 02 19:28:04.190450 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.190412 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k8shl" event={"ID":"84e7c05f-e63a-4829-ab28-c25bf61f9193","Type":"ContainerStarted","Data":"8e2618012d39498fbbec12c20ad69aed75c62d0876ecfb88e2b2811e3b0b5fab"} Nov 02 19:28:04.192674 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:04.192801 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:04.192801 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:04.192883 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:04.192883 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:04.192963 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:04.192963 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:04.193036 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.192984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:04.193160 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193139 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.193120602 +0000 UTC m=+34.688499116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:04.193325 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193293 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:04.193392 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193349 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.193339182 +0000 UTC m=+34.688717682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:04.193666 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193649 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:04.193775 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193669 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:04.193775 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193707 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.193693636 +0000 UTC m=+34.689072137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:04.193775 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193773 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:04.193937 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193781 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:04.193937 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193802 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.193794726 +0000 UTC m=+34.689173226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:04.193937 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193848 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:04.193937 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193866 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.193860651 +0000 UTC m=+34.689239150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:04.193937 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193907 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:04.194171 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.193951242 +0000 UTC m=+34.689329741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:04.194171 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.193992 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:04.194171 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.194010 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.194004902 +0000 UTC m=+34.689383403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:04.194171 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.194043 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:04.194171 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:04.194070 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:06.194061006 +0000 UTC m=+34.689439513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:04.194856 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.194826 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-txxgt" event={"ID":"a3ff325f-d384-407e-af38-2c876d9b686a","Type":"ContainerStarted","Data":"52d296f691dd934bf7942cbed96dbd12ba7bedfc613507dc345bb85cceed0ccc"} Nov 02 19:28:04.194917 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.194877 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-txxgt" event={"ID":"a3ff325f-d384-407e-af38-2c876d9b686a","Type":"ContainerStarted","Data":"5e7244810ec14b9b66e58ce0518dce2254ad5a1b9b2f84b8de5a162c29e21f28"} Nov 02 19:28:04.196789 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.196737 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpjl9" event={"ID":"189ded48-fc9b-4109-828d-3c3cae8b5242","Type":"ContainerStarted","Data":"87c22c1e5b4990baab7426a46d15134e8cfc92138d38f5235d2a97efc1a0ca07"} Nov 02 19:28:04.199219 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.199194 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-52pkd" event={"ID":"85cc6f7e-1889-40bb-a184-2138db4d29ba","Type":"ContainerStarted","Data":"cd7a99a765b1c10b6f2fc4fb7e2cf82d83ccb3ddd461cbdc78d7b41645e1eac0"} Nov 02 19:28:04.199333 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.199243 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-52pkd" event={"ID":"85cc6f7e-1889-40bb-a184-2138db4d29ba","Type":"ContainerStarted","Data":"228f642268f3ba6617452775130c97b2227f551091f8516a1d2f92cb013d9b2e"} Nov 02 19:28:04.200711 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.200681 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" event={"ID":"cb731f5a-7ec8-4440-b4b9-2ada95e8d40b","Type":"ContainerStarted","Data":"03b436b1666a8957c1c82f9d9cdb3a2ae86c2aa3111fa67e71ab07930c229336"} Nov 02 19:28:04.203532 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.203464 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w8zlt" event={"ID":"f16e19f1-3da4-43db-b861-7146aa9dbede","Type":"ContainerStarted","Data":"a780a4583a03dc51d5ff9a18f01711d63109e4f1e81debbcb77872106b61c1ff"} Nov 02 19:28:04.619274 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.619209 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-txxgt" podStartSLOduration=459.61918877 podStartE2EDuration="7m39.61918877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:04.617560819 +0000 UTC m=+33.112939342" watchObservedRunningTime="2026-11-02 19:28:04.61918877 +0000 UTC m=+33.114567294" Nov 02 19:28:04.731337 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.731247 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-52pkd" podStartSLOduration=459.731233566 podStartE2EDuration="7m39.731233566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:04.730668691 +0000 UTC m=+33.226047215" watchObservedRunningTime="2026-11-02 19:28:04.731233566 +0000 UTC m=+33.226612084" Nov 02 19:28:04.849354 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:04.849241 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k8shl" podStartSLOduration=459.849220447 podStartE2EDuration="7m39.849220447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:04.847901368 +0000 UTC m=+33.343279892" watchObservedRunningTime="2026-11-02 19:28:04.849220447 +0000 UTC m=+33.344598973" Nov 02 19:28:05.912839 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:05.912225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:05.912839 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:05.912351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:05.912839 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:05.912650 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 19:28:05.912839 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:05.912727 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:28:37.912705591 +0000 UTC m=+66.408084108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : secret "metrics-daemon-secret" not found Nov 02 19:28:05.916057 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:05.915967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:06.013000 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.012964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:06.020692 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.020659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:06.103186 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.103109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:06.113344 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.113155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:06.214347 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:06.214347 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:06.214553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:06.214553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:06.214553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:06.214553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:06.214553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:06.214553 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.214544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:06.214797 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.214670 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:06.214797 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.214729 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.214711412 +0000 UTC m=+38.710089926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215727 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215771 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.215758023 +0000 UTC m=+38.711136535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215818 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.215810338 +0000 UTC m=+38.711188851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215851 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215869 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.215863831 +0000 UTC m=+38.711242344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215907 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215913 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215932 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.215926636 +0000 UTC m=+38.711305149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215963 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215969 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.215986 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.215979296 +0000 UTC m=+38.711357808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.216020 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.216037 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.216031635 +0000 UTC m=+38.711410148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.216068 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:06.220574 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:06.216085 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.216080376 +0000 UTC m=+38.711458889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:06.221658 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.220829 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" event={"ID":"b5c10760-26db-48ed-8fe8-92700a8fcf06","Type":"ContainerStarted","Data":"357b24e931901d25b2e4e9ccd6aca317123d4a90e41a2c2f618c1e010a4e4f16"} Nov 02 19:28:06.336752 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.336694 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" podStartSLOduration=432.913524995 podStartE2EDuration="7m15.336675443s" totalImagesPullingTime="2.423150448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:06.335129363 +0000 UTC m=+34.830507885" watchObservedRunningTime="2026-11-02 19:28:06.336675443 +0000 UTC m=+34.832053970" Nov 02 19:28:06.484894 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.484857 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2dw9d"] Nov 02 19:28:06.522787 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:06.521614 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea7ce1f_d9c4_40b4_a160_0f7788c416a6.slice/crio-0bdb440db9bdcc2233377bd5611a99a3b72e276e2827592b71d3c6811fd2fff6 WatchSource:0}: Error finding container 0bdb440db9bdcc2233377bd5611a99a3b72e276e2827592b71d3c6811fd2fff6: Status 404 returned error can't find the container with id 0bdb440db9bdcc2233377bd5611a99a3b72e276e2827592b71d3c6811fd2fff6 Nov 02 19:28:06.531544 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:06.531515 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wmg7k"] Nov 02 19:28:06.531667 ip-10-0-132-174 kubenswrapper[2578]: W1102 19:28:06.531542 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdecbd0_dcd5_42f0_85ce_57b32e400992.slice/crio-980055b40e485a207a302ffa51b3d739d54bff56247df5bc671c941e81aa85cb WatchSource:0}: Error finding container 980055b40e485a207a302ffa51b3d739d54bff56247df5bc671c941e81aa85cb: Status 404 returned error can't find the container with id 980055b40e485a207a302ffa51b3d739d54bff56247df5bc671c941e81aa85cb Nov 02 19:28:07.227068 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:07.227028 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2dw9d" event={"ID":"fea7ce1f-d9c4-40b4-a160-0f7788c416a6","Type":"ContainerStarted","Data":"0bdb440db9bdcc2233377bd5611a99a3b72e276e2827592b71d3c6811fd2fff6"} Nov 02 19:28:07.231762 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:07.231694 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmg7k" event={"ID":"1fdecbd0-dcd5-42f0-85ce-57b32e400992","Type":"ContainerStarted","Data":"980055b40e485a207a302ffa51b3d739d54bff56247df5bc671c941e81aa85cb"} Nov 02 19:28:09.241012 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.240962 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpjl9" event={"ID":"189ded48-fc9b-4109-828d-3c3cae8b5242","Type":"ContainerStarted","Data":"3ac8c2fe330c04b916cafbebc486984c87c0652996a8be65578cbf8adba27ec0"} Nov 02 19:28:09.243243 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.243214 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" event={"ID":"add7fff3-f01d-4f58-893f-7cf024ae59e2","Type":"ContainerStarted","Data":"8d6089fdb2e1970fb06cb44ec668dfaafa745a6aeeb50050d8c204802827d1be"} Nov 02 19:28:09.246767 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.246742 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/0.log" Nov 02 19:28:09.246887 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.246785 2578 generic.go:356] "Generic (PLEG): container finished" podID="1804f9cc-5858-4cbb-a623-af98ee3234c2" containerID="a28fa7d661317a7d5d1bc4662c66b332ec8172ad329734d229c7536fc92acaa2" exitCode=255 Nov 02 19:28:09.247030 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.247009 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" event={"ID":"1804f9cc-5858-4cbb-a623-af98ee3234c2","Type":"ContainerDied","Data":"a28fa7d661317a7d5d1bc4662c66b332ec8172ad329734d229c7536fc92acaa2"} Nov 02 19:28:09.247114 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.247101 2578 scope.go:122] "RemoveContainer" containerID="a28fa7d661317a7d5d1bc4662c66b332ec8172ad329734d229c7536fc92acaa2" Nov 02 19:28:09.250322 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.250275 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" event={"ID":"be54cb41-84fc-4d14-a8c8-0435cf0954c9","Type":"ContainerStarted","Data":"fe085febe6eba893d886b8efd1192ba33f322b6637a532d1a4d39a319fdc1360"} Nov 02 19:28:09.252659 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.252631 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w8zlt" event={"ID":"f16e19f1-3da4-43db-b861-7146aa9dbede","Type":"ContainerStarted","Data":"848ffa46cc1a7a438e7a5fac7276960ea3de60cfd7d910c848bed1e634d79ad0"} Nov 02 19:28:09.360323 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.360214 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vpjl9" podStartSLOduration=1.771536555 podStartE2EDuration="7.36019394s" totalImagesPullingTime="5.588657385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:09.354085221 +0000 UTC m=+37.849463744" watchObservedRunningTime="2026-11-02 19:28:09.36019394 +0000 UTC m=+37.855572462" Nov 02 19:28:09.469642 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.469588 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" podStartSLOduration=433.143133764 podStartE2EDuration="7m18.469575414s" totalImagesPullingTime="5.32644165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:09.467636313 +0000 UTC m=+37.963014836" watchObservedRunningTime="2026-11-02 19:28:09.469575414 +0000 UTC m=+37.964953936" Nov 02 19:28:09.702285 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.702174 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" podStartSLOduration=433.080419516 podStartE2EDuration="7m18.702154297s" totalImagesPullingTime="5.621734781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:09.701611291 +0000 UTC m=+38.196989850" watchObservedRunningTime="2026-11-02 19:28:09.702154297 +0000 UTC m=+38.197532851" Nov 02 19:28:09.902030 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.899789 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-w8zlt" podStartSLOduration=433.847090591 podStartE2EDuration="7m18.899769133s" totalImagesPullingTime="5.052678542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:09.816070951 +0000 UTC m=+38.311449476" watchObservedRunningTime="2026-11-02 19:28:09.899769133 +0000 UTC m=+38.395147652" Nov 02 19:28:09.902030 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.901068 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fb2w5"] Nov 02 19:28:09.909180 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.909102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:09.962009 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.961719 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:28:09.962233 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.962211 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k9tfs\"" Nov 02 19:28:09.962233 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.962219 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:28:09.964188 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.964166 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fb2w5"] Nov 02 19:28:09.968124 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.967750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-crio-socket\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:09.968124 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.967806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:09.968124 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.967825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:09.968124 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.967857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-data-volume\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:09.968124 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:09.967992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcmxg\" (UniqueName: \"kubernetes.io/projected/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-api-access-gcmxg\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.068939 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.068902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcmxg\" (UniqueName: \"kubernetes.io/projected/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-api-access-gcmxg\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.069118 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.069000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-crio-socket\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.069118 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.069070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.069118 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.069093 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-crio-socket\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.069250 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.069099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.069250 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.069170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-data-volume\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.069250 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.069175 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:10.069420 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.069218 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:28:10.069420 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.069323 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.569283417 +0000 UTC m=+39.064661931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:10.069420 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.069355 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:10.56933639 +0000 UTC m=+39.064714908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : configmap "kube-rbac-proxy" not found Nov 02 19:28:10.069543 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.069521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-data-volume\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.128100 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.128070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcmxg\" (UniqueName: \"kubernetes.io/projected/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-api-access-gcmxg\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.258166 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.258133 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/1.log" Nov 02 19:28:10.258589 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.258574 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/0.log" Nov 02 19:28:10.258627 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.258610 2578 generic.go:356] "Generic (PLEG): container finished" podID="1804f9cc-5858-4cbb-a623-af98ee3234c2" containerID="355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395" exitCode=255 Nov 02 19:28:10.258728 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.258697 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" event={"ID":"1804f9cc-5858-4cbb-a623-af98ee3234c2","Type":"ContainerDied","Data":"355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395"} Nov 02 19:28:10.258858 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.258768 2578 scope.go:122] "RemoveContainer" containerID="a28fa7d661317a7d5d1bc4662c66b332ec8172ad329734d229c7536fc92acaa2" Nov 02 19:28:10.259017 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.259004 2578 scope.go:122] "RemoveContainer" containerID="355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395" Nov 02 19:28:10.259207 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.259189 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:10.261753 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.261050 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" event={"ID":"cb731f5a-7ec8-4440-b4b9-2ada95e8d40b","Type":"ContainerStarted","Data":"d2d55f914b0126105861a330ace5fa01d35ab7b67ebb406b2e6351de61f66191"} Nov 02 19:28:10.271613 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.271585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:10.271695 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.271629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:10.271695 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.271662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:10.271777 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.271703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:10.271777 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271713 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:10.271777 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271730 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:10.271777 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.271731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271778 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271793 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271819 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271834 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271784 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.27176637 +0000 UTC m=+46.767144886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271783 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271907 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.271895204 +0000 UTC m=+46.767273706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271921 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.271915861 +0000 UTC m=+46.767294361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271931 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.271926292 +0000 UTC m=+46.767304791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:10.271938 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.271943 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.271938076 +0000 UTC m=+46.767316576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.271962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.271989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.272049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.272069 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.272107 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.272096322 +0000 UTC m=+46.767474827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.272135 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.272163 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.272146486 +0000 UTC m=+46.767525006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:10.272323 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.272187 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:18.272175744 +0000 UTC m=+46.767554250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:10.529692 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.528738 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" podStartSLOduration=429.005541084 podStartE2EDuration="7m15.528717789s" totalImagesPullingTime="6.523176705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:10.527622825 +0000 UTC m=+39.023001373" watchObservedRunningTime="2026-11-02 19:28:10.528717789 +0000 UTC m=+39.024096312" Nov 02 19:28:10.552964 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.552928 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f"] Nov 02 19:28:10.571908 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.571880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" Nov 02 19:28:10.575465 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.575433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.575607 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.575474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:10.575607 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.575579 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:28:10.575708 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.575648 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:11.575626567 +0000 UTC m=+40.071005068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : configmap "kube-rbac-proxy" not found Nov 02 19:28:10.575770 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.575708 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:10.575770 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:10.575765 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:11.575750557 +0000 UTC m=+40.071129071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:10.611156 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.611040 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f"] Nov 02 19:28:10.624446 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.624418 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m9rmj\"" Nov 02 19:28:10.624446 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.624419 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:10.624607 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.624427 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 19:28:10.677040 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.676996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbn4h\" (UniqueName: \"kubernetes.io/projected/24a6a6ac-45b7-4d97-9a4d-9f74db384832-kube-api-access-gbn4h\") pod \"migrator-66df5d496b-fzv6f\" (UID: \"24a6a6ac-45b7-4d97-9a4d-9f74db384832\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" Nov 02 19:28:10.777723 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.777692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbn4h\" (UniqueName: \"kubernetes.io/projected/24a6a6ac-45b7-4d97-9a4d-9f74db384832-kube-api-access-gbn4h\") pod \"migrator-66df5d496b-fzv6f\" (UID: \"24a6a6ac-45b7-4d97-9a4d-9f74db384832\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" Nov 02 19:28:10.835504 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.835431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbn4h\" (UniqueName: \"kubernetes.io/projected/24a6a6ac-45b7-4d97-9a4d-9f74db384832-kube-api-access-gbn4h\") pod \"migrator-66df5d496b-fzv6f\" (UID: \"24a6a6ac-45b7-4d97-9a4d-9f74db384832\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" Nov 02 19:28:10.891202 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:10.891176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" Nov 02 19:28:11.265835 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.265797 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wmg7k" event={"ID":"1fdecbd0-dcd5-42f0-85ce-57b32e400992","Type":"ContainerStarted","Data":"50c547283754ad6b001d1ef648d5c3a34e398c1a49cc3a590fe2999a784cce87"} Nov 02 19:28:11.266346 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.265862 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:11.267406 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.267380 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/1.log" Nov 02 19:28:11.267678 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.267664 2578 scope.go:122] "RemoveContainer" containerID="355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395" Nov 02 19:28:11.267849 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:11.267833 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:11.291161 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.291139 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f"] Nov 02 19:28:11.381835 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.381787 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wmg7k" podStartSLOduration=35.027358301 podStartE2EDuration="39.38177406s" totalImagesPullingTime="4.354415759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:11.381113259 +0000 UTC m=+39.876491782" watchObservedRunningTime="2026-11-02 19:28:11.38177406 +0000 UTC m=+39.877152582" Nov 02 19:28:11.586879 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.586799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:11.586879 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:11.586833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:11.587089 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:11.586937 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:28:11.587089 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:11.587020 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:13.58699838 +0000 UTC m=+42.082376900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : configmap "kube-rbac-proxy" not found Nov 02 19:28:11.587089 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:11.586951 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:11.587229 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:11.587095 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:13.587079826 +0000 UTC m=+42.082458341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:12.272753 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:12.272718 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" event={"ID":"24a6a6ac-45b7-4d97-9a4d-9f74db384832","Type":"ContainerStarted","Data":"a880088f3ef0a752e49f81be75567413272847828acf1ba597a5883a391bc8cd"} Nov 02 19:28:12.780181 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:12.780137 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:12.780399 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:12.780218 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:12.780648 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:12.780625 2578 scope.go:122] "RemoveContainer" containerID="355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395" Nov 02 19:28:12.780911 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:12.780875 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:13.608587 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:13.608551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:13.608942 ip-10-0-132-174 kubenswrapper[2578]: I1102 19:28:13.608590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:13.608942 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:13.608687 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:28:13.608942 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:13.608709 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:13.608942 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:13.608754 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.608741 +0000 UTC m=+46.104119513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : configmap "kube-rbac-proxy" not found Nov 02 19:28:13.608942 ip-10-0-132-174 kubenswrapper[2578]: E1102 19:28:13.608770 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.608764733 +0000 UTC m=+46.104143246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:13.977432 ip-10-0-132-174 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 19:28:13.991717 ip-10-0-132-174 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 19:28:13.991880 ip-10-0-132-174 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 19:28:13.991931 ip-10-0-132-174 systemd[1]: kubelet.service: Consumed 3.069s CPU time. Nov 02 19:28:14.003039 ip-10-0-132-174 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:28:14.100502 ip-10-0-132-174 kubenswrapper[4966]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:28:14.100502 ip-10-0-132-174 kubenswrapper[4966]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:28:14.100502 ip-10-0-132-174 kubenswrapper[4966]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:28:14.100502 ip-10-0-132-174 kubenswrapper[4966]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:28:14.102525 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102444 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.102525 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102525 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.102525 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102529 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102533 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102536 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102538 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102541 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102544 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102547 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102549 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102552 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102555 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102558 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102560 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102563 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102566 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102568 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102571 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102573 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102576 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102578 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102581 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.102634 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102584 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102587 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102595 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102599 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102601 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102604 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102607 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102610 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102612 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102615 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102617 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102620 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102623 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102626 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102628 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102631 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102634 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102636 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102639 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.103137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102642 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102645 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102648 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102650 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102653 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102655 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102658 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102661 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102663 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102666 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102668 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102671 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102673 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102676 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102678 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102682 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102684 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102687 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102690 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.103626 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102692 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102695 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102697 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102700 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102702 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102705 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102709 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102712 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102715 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102718 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102722 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102725 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102729 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102732 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102735 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102739 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102742 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102744 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102747 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.104113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102750 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102752 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102755 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102757 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102760 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102762 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102765 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102768 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102770 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102772 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102777 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102780 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102783 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102786 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102790 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102793 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102795 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102798 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102800 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.104615 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102803 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102806 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102808 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102811 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102813 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102816 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102818 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.102821 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103568 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103575 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103578 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103581 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103584 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103588 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103590 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103593 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103595 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103598 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103601 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103603 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.105137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103606 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103608 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103611 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103614 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103616 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103619 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103621 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103626 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103629 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103632 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103634 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103637 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103639 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103642 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103645 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103647 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103650 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103652 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103655 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.105706 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103660 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103662 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103665 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103667 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103671 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103673 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103676 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103679 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103681 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103684 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103686 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103689 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103691 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103694 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103697 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103699 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103702 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103704 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103707 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.106203 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103709 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103712 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103715 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103718 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103721 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103723 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103728 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103731 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103734 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103737 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103740 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103742 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103745 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103750 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103753 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103755 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103758 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103760 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103764 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103767 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.106697 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103769 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103772 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103775 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103777 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103780 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103782 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103785 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103787 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103790 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103793 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103795 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103798 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103801 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103803 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103806 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103809 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103812 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103815 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103817 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.107207 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103820 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103824 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103827 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103830 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103833 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103835 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103839 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103842 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103844 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103847 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103850 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103853 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103856 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103859 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103862 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103865 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.103867 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104389 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104395 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104398 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.107689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104401 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104404 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104406 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104409 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104412 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104414 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104417 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104420 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104423 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104425 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104428 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104430 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104433 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104436 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104438 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104441 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104444 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104446 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104449 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.108172 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104452 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104454 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104457 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104459 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104462 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104465 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104468 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104471 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104473 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104477 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104481 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104483 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104486 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104489 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104491 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104494 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104496 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104499 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104502 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.108663 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104505 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104508 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104510 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104513 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104515 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104518 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104521 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104524 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104527 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104529 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104533 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104535 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104538 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104540 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104543 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104545 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104548 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104550 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104553 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.109137 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104556 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104559 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104561 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104564 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104566 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104569 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104571 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104574 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104577 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104579 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104582 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104584 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104587 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104590 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104592 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104595 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104597 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104600 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104602 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104605 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.109650 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104608 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104610 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104613 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104615 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104618 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104620 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104623 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104626 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104628 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104631 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104634 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104636 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104639 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104641 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104644 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104648 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104652 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104655 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104657 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.110171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104661 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104663 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104666 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104669 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104671 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104674 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.104676 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104773 4966 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104780 4966 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104789 4966 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104796 4966 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104802 4966 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104807 4966 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104813 4966 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104817 4966 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104821 4966 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104824 4966 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104827 4966 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104830 4966 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104834 4966 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104836 4966 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104839 4966 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104845 4966 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:28:14.110672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104848 4966 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104851 4966 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104855 4966 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104858 4966 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104862 4966 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104865 4966 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104869 4966 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104874 4966 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104877 4966 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104880 4966 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104884 4966 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104887 4966 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104891 4966 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104894 4966 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104897 4966 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104899 4966 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104904 4966 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104908 4966 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104910 4966 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104913 4966 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104916 4966 flags.go:64] FLAG: --enable-server="true" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104919 4966 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104924 4966 flags.go:64] FLAG: --event-burst="100" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104927 4966 flags.go:64] FLAG: --event-qps="50" Nov 02 19:28:14.111249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104930 4966 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104933 4966 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104936 4966 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104940 4966 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104943 4966 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104946 4966 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104949 4966 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104955 4966 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104958 4966 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104961 4966 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104965 4966 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104968 4966 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104971 4966 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104974 4966 flags.go:64] FLAG: --feature-gates="" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104978 4966 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104983 4966 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104986 4966 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104989 4966 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104993 4966 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104996 4966 flags.go:64] FLAG: --help="false" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.104999 4966 flags.go:64] FLAG: --hostname-override="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105002 4966 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105005 4966 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:28:14.111889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105008 4966 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105012 4966 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105015 4966 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105018 4966 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105021 4966 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105024 4966 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105027 4966 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105029 4966 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105033 4966 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105036 4966 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105038 4966 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105042 4966 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105045 4966 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105047 4966 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105050 4966 flags.go:64] FLAG: --lock-file="" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105053 4966 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105057 4966 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105061 4966 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105066 4966 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105069 4966 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105072 4966 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105075 4966 flags.go:64] FLAG: --logging-format="text" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105078 4966 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105082 4966 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:28:14.112511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105087 4966 flags.go:64] FLAG: --manifest-url="" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105090 4966 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105094 4966 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105097 4966 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105101 4966 flags.go:64] FLAG: --max-pods="110" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105104 4966 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105107 4966 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105110 4966 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105114 4966 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105117 4966 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105120 4966 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105123 4966 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105130 4966 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105133 4966 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105136 4966 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105140 4966 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105143 4966 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105146 4966 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105150 4966 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105153 4966 flags.go:64] FLAG: --port="10250" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105156 4966 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105159 4966 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0010e02df6a54c2de" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105162 4966 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105165 4966 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105169 4966 flags.go:64] FLAG: --register-node="true" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105172 4966 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:28:14.113174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105176 4966 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105180 4966 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105182 4966 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105186 4966 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105190 4966 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105193 4966 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105197 4966 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105201 4966 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105204 4966 flags.go:64] FLAG: --runonce="false" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105207 4966 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105210 4966 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105213 4966 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105215 4966 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105219 4966 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105222 4966 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105225 4966 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105228 4966 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105231 4966 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105234 4966 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105237 4966 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105240 4966 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105243 4966 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105246 4966 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105249 4966 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105255 4966 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:28:14.114261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105258 4966 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105261 4966 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105266 4966 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105269 4966 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105272 4966 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105276 4966 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105279 4966 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105282 4966 flags.go:64] FLAG: --v="2" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105286 4966 flags.go:64] FLAG: --version="false" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105306 4966 flags.go:64] FLAG: --vmodule="" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105311 4966 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105314 4966 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105469 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105475 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105479 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105482 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105485 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105489 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105492 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105494 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105497 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105500 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105503 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.115046 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105506 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105508 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105511 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105513 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105516 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105519 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105521 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105524 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105527 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105529 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105532 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105535 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105537 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105540 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105544 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105546 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105549 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105552 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105554 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.115649 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105557 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105559 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105562 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105566 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105569 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105572 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105575 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105577 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105580 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105583 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105585 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105588 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105591 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105593 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105596 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105598 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105601 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105603 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105606 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105609 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.116171 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105611 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105614 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105616 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105619 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105621 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105624 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105626 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105631 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105633 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105636 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105639 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105641 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105645 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105649 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105652 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105657 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105660 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105663 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105666 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.116764 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105669 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105672 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105675 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105677 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105680 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105682 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105685 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105687 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105690 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105693 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105695 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105699 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105703 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105707 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105710 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105713 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105715 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105718 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105721 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.117417 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105724 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105728 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105730 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105733 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105736 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105739 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105748 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105751 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105754 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105757 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105760 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105763 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105765 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105768 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105771 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105774 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105776 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.118279 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105779 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.105806 4966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105840 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105844 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105847 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105850 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105853 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105856 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105859 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105861 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105864 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105867 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105869 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105872 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105875 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.118961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105878 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105880 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105884 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105887 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105890 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105892 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105895 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105898 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105900 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105903 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105906 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105908 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105911 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105914 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105917 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105919 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105922 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105925 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105928 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105930 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.119415 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105933 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105935 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105938 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105941 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105943 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105945 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105948 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105950 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105953 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105955 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105958 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105960 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105963 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105966 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105969 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105972 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105975 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105977 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105980 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.119992 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105983 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105985 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105988 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105990 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105993 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105996 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.105998 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106001 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106004 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106006 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106009 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106011 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106014 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106017 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106020 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106022 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106025 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106027 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106030 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106033 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.120478 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106035 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106038 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106040 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106043 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106045 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106048 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106050 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106054 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106057 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106059 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106062 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106065 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106067 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106070 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106073 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106077 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106080 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106083 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106086 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.120965 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106088 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106091 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106094 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106097 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106099 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106102 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106105 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106107 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106110 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106112 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106115 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106117 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106120 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106122 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.106127 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.109839 4966 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.109857 4966 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.109877 4966 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.109881 4966 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:28:14.121449 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109941 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109945 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109949 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109952 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109955 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109957 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109960 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109963 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109966 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109969 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109972 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109975 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109978 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109980 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109983 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109986 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109988 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109991 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109993 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109996 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.121923 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.109999 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110001 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110004 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110007 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110010 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110013 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110016 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110019 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110022 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110025 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110028 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110030 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110033 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110035 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110038 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110041 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110043 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110046 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110048 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110051 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.122426 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110054 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110056 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110059 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110061 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110064 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110067 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110069 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110072 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110074 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110078 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110082 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110085 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110088 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110091 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110094 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110097 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110099 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110102 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110104 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110107 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.122932 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110110 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110114 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110131 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110134 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110138 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110141 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110144 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110146 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110149 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110152 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110154 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110157 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110159 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110163 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110165 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110168 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110170 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110173 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110176 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.123457 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110178 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110181 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110183 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110186 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110189 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110191 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110194 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110196 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110199 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110201 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110204 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110206 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110209 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110212 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110215 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110217 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110220 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110223 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110225 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.123913 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110228 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110230 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110233 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110237 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110241 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110244 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110247 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110250 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.110277 4966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110431 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110436 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110439 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110441 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110444 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110447 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.124410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110449 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110452 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110454 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110457 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110459 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110462 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110465 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110468 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110473 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110477 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110480 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110483 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110486 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110489 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110492 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110495 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110497 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110500 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110502 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.124780 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110505 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110507 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110510 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110512 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110515 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110518 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110520 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110523 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110525 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110528 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110530 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110533 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110535 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110538 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110540 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110543 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110546 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110548 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110551 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.125283 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110555 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110558 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110562 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110565 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110568 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110570 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110573 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110576 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110578 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110581 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110583 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110586 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110588 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110591 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110594 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110596 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110599 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110601 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110604 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110606 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.125769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110609 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110611 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110614 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110616 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110619 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110621 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110624 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110626 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110629 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110631 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110634 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110637 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110639 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110641 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110644 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110646 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110649 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110652 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110654 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110657 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.126257 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110660 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110662 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110665 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110667 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110670 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110672 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110675 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110678 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110680 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110683 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110689 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110691 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110694 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110697 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110699 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110702 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110704 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110707 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110709 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110711 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.126815 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110714 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110717 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.110740 4966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110771 4966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110776 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110779 4966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110782 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110785 4966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110787 4966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110791 4966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110793 4966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110796 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110798 4966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110802 4966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110804 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110807 4966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:28:14.127460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110809 4966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110812 4966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110814 4966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110817 4966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110820 4966 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110822 4966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110826 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110829 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110831 4966 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110834 4966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110837 4966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110840 4966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110842 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110845 4966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110847 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110850 4966 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110852 4966 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110855 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110857 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110860 4966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:28:14.127886 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110862 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110865 4966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110867 4966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110870 4966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110872 4966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110875 4966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110877 4966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110880 4966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110882 4966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110885 4966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110888 4966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110891 4966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110893 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110896 4966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110899 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110901 4966 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110904 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110906 4966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110911 4966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110914 4966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:28:14.128386 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110917 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110919 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110922 4966 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110924 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110926 4966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110929 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110932 4966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110934 4966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110937 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110940 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110942 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110945 4966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110947 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110950 4966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110953 4966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110957 4966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110961 4966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110964 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110967 4966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:28:14.128870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110970 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110973 4966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110977 4966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110980 4966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110983 4966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110986 4966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110988 4966 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110991 4966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110993 4966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110996 4966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.110998 4966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111002 4966 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111005 4966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111007 4966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111010 4966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111012 4966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111015 4966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111017 4966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111020 4966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:28:14.129394 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111022 4966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111025 4966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111027 4966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111030 4966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111033 4966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111035 4966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111038 4966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111041 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111045 4966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111049 4966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111052 4966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111054 4966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111057 4966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111059 4966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:14.111062 4966 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.111120 4966 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.112489 4966 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.112540 4966 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.112732 4966 server.go:1014] "Starting client certificate rotation" Nov 02 19:28:14.129870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.112868 4966 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.113157 4966 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:22:31 +0000 UTC" deadline="2028-05-24 10:00:00.029025909 +0000 UTC" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.113239 4966 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13646h31m45.915791278s" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.113786 4966 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.115524 4966 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.116711 4966 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.118758 4966 log.go:25] "Validated CRI v1 image API" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.119479 4966 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:28:14.130393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.121366 4966 fs.go:135] Filesystem UUIDs: map[1c016816-0d4b-4289-977d-b35f16ba7215:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b8ab3d44-23a2-4bc4-8c96-e3f75df8e0b4:/dev/nvme0n1p4] Nov 02 19:28:14.130615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.121383 4966 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03b436b1666a8957c1c82f9d9cdb3a2ae86c2aa3111fa67e71ab07930c229336/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03b436b1666a8957c1c82f9d9cdb3a2ae86c2aa3111fa67e71ab07930c229336/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bdb440db9bdcc2233377bd5611a99a3b72e276e2827592b71d3c6811fd2fff6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bdb440db9bdcc2233377bd5611a99a3b72e276e2827592b71d3c6811fd2fff6/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e4b6773962956f2d1372da9377a42524ec23f86a2bbe93dd6d5f66a74819134/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e4b6773962956f2d1372da9377a42524ec23f86a2bbe93dd6d5f66a74819134/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fcc1c87c926650c79308f45aed6ce979b3e78eeee4bc1763a49139452f6d65d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fcc1c87c926650c79308f45aed6ce979b3e78eeee4bc1763a49139452f6d65d/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/228f642268f3ba6617452775130c97b2227f551091f8516a1d2f92cb013d9b2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/228f642268f3ba6617452775130c97b2227f551091f8516a1d2f92cb013d9b2e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26cdce1aa97ace90c9ebf3d9eb6f76b419a44958e235830ce0cf766201dbcbfe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26cdce1aa97ace90c9ebf3d9eb6f76b419a44958e235830ce0cf766201dbcbfe/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e2b0c34247ff094e999c42aedf01d2c017c032ed54df4e40ecc6e722f59cca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e2b0c34247ff094e999c42aedf01d2c017c032ed54df4e40ecc6e722f59cca7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50c6edd275cac77e2abb2a93709a893bdb350a36d624d0cc3f23589c6cab21fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50c6edd275cac77e2abb2a93709a893bdb350a36d624d0cc3f23589c6cab21fc/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/555e7d478466bbd651f8f79901a34fd0f6249269c925e5c987d40aaba9fb6729/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/555e7d478466bbd651f8f79901a34fd0f6249269c925e5c987d40aaba9fb6729/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e7244810ec14b9b66e58ce0518dce2254ad5a1b9b2f84b8de5a162c29e21f28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e7244810ec14b9b66e58ce0518dce2254ad5a1b9b2f84b8de5a162c29e21f28/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ef2d819a1b2eb5b4a1d7f164b35868b83f8e06b0ba5504e6d3d0ebe1eab9457/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ef2d819a1b2eb5b4a1d7f164b35868b83f8e06b0ba5504e6d3d0ebe1eab9457/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/671c112c9149b2bd8db675a1c244110b2d1173ed435e64b6a8ba1b33ac5d2ac7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/671c112c9149b2bd8db675a1c244110b2d1173ed435e64b6a8ba1b33ac5d2ac7/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87c22c1e5b4990baab7426a46d15134e8cfc92138d38f5235d2a97efc1a0ca07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87c22c1e5b4990baab7426a46d15134e8cfc92138d38f5235d2a97efc1a0ca07/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/980055b40e485a207a302ffa51b3d739d54bff56247df5bc671c941e81aa85cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/980055b40e485a207a302ffa51b3d739d54bff56247df5bc671c941e81aa85cb/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2ef35a9e954227445b49998d78567e5aba76885a034f4cc2c45e79fef076d48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2ef35a9e954227445b49998d78567e5aba76885a034f4cc2c45e79fef076d48/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4daae0fffba6f20b63e1dc34510996a2be78cf4bc6057ebb318e094c795d4ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4daae0fffba6f20b63e1dc34510996a2be78cf4bc6057ebb318e094c795d4ba/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a780a4583a03dc51d5ff9a18f01711d63109e4f1e81debbcb77872106b61c1ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a780a4583a03dc51d5ff9a18f01711d63109e4f1e81debbcb77872106b61c1ff/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a880088f3ef0a752e49f81be75567413272847828acf1ba597a5883a391bc8cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a880088f3ef0a752e49f81be75567413272847828acf1ba597a5883a391bc8cd/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fccae350b6edf37351f93eaa658b8873d4e6f1a8e850b6a0b66ae8b19472e4d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fccae350b6edf37351f93eaa658b8873d4e6f1a8e850b6a0b66ae8b19472e4d2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff4158ee425ebd108bb44e0dd15d482d3ca5176078dc284c3dbeeeae41083af1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff4158ee425ebd108bb44e0dd15d482d3ca5176078dc284c3dbeeeae41083af1/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff4d1280f420614c96c12e23f03cdcbace6f9b6215c873e9e623625a3e94f9e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff4d1280f420614c96c12e23f03cdcbace6f9b6215c873e9e623625a3e94f9e3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~projected/kube-api-access-r4cqc:{mountpoint:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~projected/kube-api-access-r4cqc major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1804f9cc-5858-4cbb-a623-af98ee3234c2/volumes/kubernetes.io~projected/kube-api-access-cnfpf:{mountpoint:/var/lib/kubelet/pods/1804f9cc-5858-4cbb-a623-af98ee3234c2/volumes/kubernetes.io~projected/kube-api-access-cnfpf major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1804f9cc-5858-4cbb-a623-af98ee3234c2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1804f9cc-5858-4cbb-a623-af98ee3234c2/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/189ded48-fc9b-4109-828d-3c3cae8b5242/volumes/kubernetes.io~projected/kube-api-access-s2kkk:{mountpoint:/var/lib/kubelet/pods/189ded48-fc9b-4109-828d-3c3cae8b5242/volumes/kubernetes.io~projected/kube-api-access-s2kkk major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/189ded48-fc9b-4109-828d-3c3cae8b5242/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/189ded48-fc9b-4109-828d-3c3cae8b5242/volumes/kubernetes.io~secret/cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fdecbd0-dcd5-42f0-85ce-57b32e400992/volumes/kubernetes.io~projected/kube-api-access-z8qhm:{mountpoint:/var/lib/kubelet/pods/1fdecbd0-dcd5-42f0-85ce-57b32e400992/volumes/kubernetes.io~projected/kube-api-access-z8qhm major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24a6a6ac-45b7-4d97-9a4d-9f74db384832/volumes/kubernetes.io~projected/kube-api-access-gbn4h:{mountpoint:/var/lib/kubelet/pods/24a6a6ac-45b7-4d97-9a4d-9f74db384832/volumes/kubernetes.io~projected/kube-api-access-gbn4h major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2939006c-2a64-4684-968e-dc69c92c23ea/volumes/kubernetes.io~projected/kube-api-access-z6q8h:{mountpoint:/var/lib/kubelet/pods/2939006c-2a64-4684-968e-dc69c92c23ea/volumes/kubernetes.io~projected/kube-api-access-z6q8h major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a889062-1b3c-442a-860d-64e9209a6acf/volumes/kubernetes.io~projected/kube-api-access-pr9qr:{mountpoint:/var/lib/kubelet/pods/3a889062-1b3c-442a-860d-64e9209a6acf/volumes/kubernetes.io~projected/kube-api-access-pr9qr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~projected/kube-api-access-6mr9n:{mountpoint:/var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~projected/kube-api-access-6mr9n major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52b4ca7a-3abb-4c17-a460-51f0b45cadca/volumes/kubernetes.io~projected/kube-api-access-t6z6h:{mountpoint:/var/lib/kubelet/pods/52b4ca7a-3abb-4c17-a460-51f0b45cadca/volumes/kubernetes.io~projected/kube-api-access-t6z6h major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~projected/kube-api-access-ck8jm:{mountpoint:/var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~projected/kube-api-access-ck8jm major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~projected/kube-api-access-mql6m:{mountpoint:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~projected/kube-api-access-mql6m major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82ef8e16-cb3d-4b88-9a4e-6970f544b650/volumes/kubernetes.io~projected/kube-api-access-rgzg5:{mountpoint:/var/lib/kubelet/pods/82ef8e16-cb3d-4b88-9a4e-6970f544b650/volumes/kubernetes.io~projected/kube-api-access-rgzg5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84e7c05f-e63a-4829-ab28-c25bf61f9193/volumes/kubernetes.io~projected/kube-api-access-rh5ms:{mountpoint:/var/lib/kubelet/pods/84e7c05f-e63a-4829-ab28-c25bf61f9193/volumes/kubernetes.io~projected/kube-api-access-rh5ms major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85cc6f7e-1889-40bb-a184-2138db4d29ba/volumes/kubernetes.io~projected/kube-api-access-45klt:{mountpoint:/var/lib/kubelet/pods/85cc6f7e-1889-40bb-a184-2138db4d29ba/volumes/kubernetes.io~projected/kube-api-access-45klt major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947f945b-c140-4889-aaae-d332185913c7/volumes/kubernetes.io~projected/kube-api-access-z9zfz:{mountpoint:/var/lib/kubelet/pods/947f945b-c140-4889-aaae-d332185913c7/volumes/kubernetes.io~projected/kube-api-access-z9zfz major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a9afd80-1122-46ad-8082-77f3b20738e9/volumes/kubernetes.io~projected/kube-api-access-xkgdq:{mountpoint:/var/lib/kubelet/pods/9a9afd80-1122-46ad-8082-77f3b20738e9/volumes/kubernetes.io~projected/kube-api-access-xkgdq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170/volumes/kubernetes.io~projected/kube-api-access-gcmxg:{mountpoint:/var/lib/kubelet/pods/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170/volumes/kubernetes.io~projected/kube-api-access-gcmxg major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e7785c4-f87a-4969-9d54-1aafb3c2e641/volumes/kubernetes.io~projected/kube-api-access-dl4pq:{mountpoint:/var/lib/kubelet/pods/9e7785c4-f87a-4969-9d54-1aafb3c2e641/volumes/kubernetes.io~projected/kube-api-access-dl4pq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volumes/kubernetes.io~projected/kube-api-access-6cwfh:{mountpoint:/var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volumes/kubernetes.io~projected/kube-api-access-6cwfh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ff325f-d384-407e-af38-2c876d9b686a/volumes/kubernetes.io~projected/kube-api-access-gl8qp:{mountpoint:/var/lib/kubelet/pods/a3ff325f-d384-407e-af38-2c876d9b686a/volumes/kubernetes.io~projected/kube-api-access-gl8qp major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112/volumes/kubernetes.io~projected/kube-api-access-wqv5g:{mountpoint:/var/lib/kubelet/pods/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112/volumes/kubernetes.io~projected/kube-api-access-wqv5g major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/add7fff3-f01d-4f58-893f-7cf024ae59e2/volumes/kubernetes.io~projected/kube-api-access-n4gvn:{mountpoint:/var/lib/kubelet/pods/add7fff3-f01d-4f58-893f-7cf024ae59e2/volumes/kubernetes.io~projected/kube-api-access-n4gvn major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/add7fff3-f01d-4f58-893f-7cf024ae59e2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/add7fff3-f01d-4f58-893f-7cf024ae59e2/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5c10760-26db-48ed-8fe8-92700a8fcf06/volumes/kubernetes.io~projected/kube-api-access-tr9cq:{mountpoint:/var/lib/kubelet/pods/b5c10760-26db-48ed-8fe8-92700a8fcf06/volumes/kubernetes.io~projected/kube-api-access-tr9cq major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be44fdaf-13bc-4704-8c58-bcc0359528e8/volumes/kubernetes.io~projected/kube-api-access-6czjl:{mountpoint:/var/lib/kubelet/pods/be44fdaf-13bc-4704-8c58-bcc0359528e8/volumes/kubernetes.io~projected/kube-api-access-6czjl major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be54cb41-84fc-4d14-a8c8-0435cf0954c9/volumes/kubernetes.io~projected/kube-api-access-xgfpv:{mountpoint:/var/lib/kubelet/pods/be54cb41-84fc-4d14-a8c8-0435cf0954c9/volumes/kubernetes.io~projected/kube-api-access-xgfpv major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be54cb41-84fc-4d14-a8c8-0435cf0954c9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/be54cb41-84fc-4d14-a8c8-0435cf0954c9/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4a482a0-025f-4bd6-8229-aa6971510e12/volumes/kubernetes.io~projected/kube-api-access-q9gqw:{mountpoint:/var/lib/kubelet/pods/c4a482a0-025f-4bd6-8229-aa6971510e12/volumes/kubernetes.io~projected/kube-api-access-q9gqw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b/volumes/kubernetes.io~projected/kube-api-access-4rjqh:{mountpoint:/var/lib/kubelet/pods/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b/volumes/kubernetes.io~projected/kube-api-access-4rjqh major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d14e549d-addd-40f6-b6af-12582e1a028b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d14e549d-addd-40f6-b6af-12582e1a028b/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f16e19f1-3da4-43db-b861-7146aa9dbede/volumes/kubernetes.io~projected/kube-api-access-fbzkt:{mountpoint:/var/lib/kubelet/pods/f16e19f1-3da4-43db-b861-7146aa9dbede/volumes/kubernetes.io~projected/kube-api-access-fbzkt major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f16e19f1-3da4-43db-b861-7146aa9dbede/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f16e19f1-3da4-43db-b861-7146aa9dbede/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fea7ce1f-d9c4-40b4-a160-0f7788c416a6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fea7ce1f-d9c4-40b4-a160-0f7788c416a6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f053d0620e215eddd17fd350b77aaff2817b51e6057f6e638e3703c707e0a92a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/aa8e511c9e45448d761f9ea6a0872a5c266398ec6b58130c79e9f4e3d575a3ed/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/11205a62175d4a9e2a0b1cc7ab7fe586f469bb11b9ed6eecfc49338e816ef036/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7dc58bce8fca1caf55f08c662d50b14642bece5c776a6410adadf6c3bbf51a20/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0893b388a2369ac1152a5c2d18828b72c35bb7bfec2e02a95dc27f05b7ea7c5e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e565c4debd8549d0de39abdf1351ed912759d4837126c31b569a98c0622ec721/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/716a5497a9a1300017ef06e25c2150f2653f7db5971a2abda5af9d01799785de/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fa3e416ef796f98318598d271b24078240ba7b66d556f7a4b4bdd5e7c8427c88/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/8c4c83b8c0c924dfda9e0f04a5937658f0d4b0d9a38c9b98aa1561d1133f3a9a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/ad18885879dcc9b44798c2165aaf5b2025d47937cdb47fa3cd4358e76c87ae9e/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/a4fb46f934f39ddc96dd6e5874cf816cc2d8ead10976c5723030e0f3515e266b/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/483c3ed9f6214a0df154847237e636810673faad934c0ab5c74ff0983bd1cc37/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/72c7b901fadab3142f6dc94ebd94be185ec014f1d30c641214501ada612a1c56/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fb486d590b370a793e31dc4345af2a42c60b72a73a8e51c3227f8e481729c7b6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1dcc362766b03e50c3018e9e556fdd8ae642b1b01e43228d08aa260fc1f0c99b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d354a2b426d969263ae39bc6dd3b870009d934ca163020d8681f94e8868e706c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0516b152a1d659991bfefd186715e4abadc986fe3b85d4d6f984fd84c6c18912/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3bb0cba522cf0569e9f1e9b0a53385133f8fbbce1d43694e614867d06a175c29/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c98ddfc66d83f1986b7ef3cd34b3d38f17daf1b0bcebda943c62196acd478971/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1a0119763d86df3bcd0d3732e70ff3852e37dc4b6bf3963b424d635bd116e4c9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/81c2bb0c5f6b5b66f3565c6d95942375712997708e794b6ee770993df9b403cd/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/f9d0f609cb153c5339450aee2215812f1dcb184dd5096c13940df8f3bbf4d570/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/a013a974288555bfe2f007013e21b087b9adcdc140bac8abd6b62417b42a494c/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/2049440f40fe648b37b423681d29285f8c0bdbe120857eea28578582eef5fd21/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/ff3660de8e07d9fc6eb2b150f814849fc40a90c430bc33a204d14a8050c239a7/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9e7e2e6bac9b5be5e0e1f79e58eaba29199f8ba3dd1be0b41440e8928c089545/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/d0eee8bfd3cc33b7c0e9b4217ab32155e489756755560b39210c0565570ef136/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/deae3f738789b947e2cfd721f39faff50d4a36612ad66847994cdccdf128311a/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/581faf25627df536abef5cd837e39f5f16dea049f587fa0211da97e539e7ae26/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/b897410c7a7b00e2a1054af28862cfd0a6361c10b48175f44188070edddbadec/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/573577bff2e16ed1b83fec15a25760d025553153a70170731701ff0f0470b854/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/a2556b4524f21b5f1b1ff2dc3df2886f4d8a0b905d1c691f91a2a0ee10a191e4/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/2e6b47498c87d66eddc391edcce3a257177a2ebd7a6539504835f32c4d538454/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/409ad0868f1119e3bda16aa431600546460550d3bb8ac84303ea6e85b1cd60c4/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/9f0e82860970900b1db135ad2fb5000f192c1a3f0f671aa51dd9fd25fc6568ea/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/25ce4cad135e52b8bcf65269cbdefe00425a0ce7cb0a8343925b4a36046fe059/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/40956d5eb8da349cadcca2bb50ffbcf38f315ed2b798315e48b420c37ac9ec7a/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/17476ff0c5f96b64c5375317d6650c987a6cbb85019b95bed4f1c851c2031653/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/37b20d553aff3f846a51b4fbcc43edccb4d694502c512da8085dbd574e84484c/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/2216c9932ab02c00916f2ed28e2f4012eb4011eca0417fe4d3764d4c69a1e52e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/d47f04f64b1a947dee55ab9b2677f4d916ba1b0661dc43f6a832f1e4cb0c0c63/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a2e39ac8b0d5d49c27af73841aa3be8d9c14c431d7cbe0e8c9a312377a4136c3/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/1591b1a49ba3803efdfbdac9233304a11672325b4a0e88160ba4876e4b6dcc2c/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/e253c72a87c54e90238043ca2062fc1d97cba2e4ef9487c057526e1a2c5a79c7/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/a336ee66d9849277579ac53db17c469dd7780bc5c9639c876bc25a31be2050d0/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/5246042f9076d020eabea1db49846a267cdf094221019dfecdff1d2bc2ae650d/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/c6c1d4a6a5e77a4a6675d6ae3b251dc10b31995f3f9ef8e5bbae5b4d5bffd786/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/05f6e6b493cbadeb37c55acfcea11138204ed915c0d65ff86d2ce9e48b1fe283/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/39b1d4262bf4607977db89d6c06e256c2bd891fac4c91034036ca1293a6fd895/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/baf23cc74309a113109441f5318bfca3eb5feed07e1ecdc190e7ffa6dd6e5468/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0d395d6f448d3fad3fdc7b32017241eb3be3914264b9c65e6b04220296cae427/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7813904e2751a2147dc87b184d48ba4364c16aea4005c9a936e047d9a2be2608/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/1ebda88e5fc3d86ea12a2941c3b479d0ba79aa3bd91f2475c2de11cb6b3ceffe/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3e88ff2217bf0197cff20b9ad8bcb4114761944ee204c34b678909dab9f18395/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6af4d50575be2b8d177debd7387d168deaa1e258de1155986e3d3c3cd216ddb1/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/31d71e9b47274db502a0abc5b2e2e775d813ffd602295c8b8d6ab8a6339b59b7/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.129515 4966 manager.go:217] Machine: {Timestamp:2026-11-02 19:28:14.129031244 +0000 UTC m=+0.077687076 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100318 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ada901e446db8cf6ce7dfb87c149 SystemUUID:ec23ada9-01e4-46db-8cf6-ce7dfb87c149 BootID:018a44d2-dc6b-49a9-b4d2-f99beafbcc1f Filesystems:[{Device:/var/lib/kubelet/pods/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112/volumes/kubernetes.io~projected/kube-api-access-wqv5g DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/555e7d478466bbd651f8f79901a34fd0f6249269c925e5c987d40aaba9fb6729/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff4158ee425ebd108bb44e0dd15d482d3ca5176078dc284c3dbeeeae41083af1/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50c6edd275cac77e2abb2a93709a893bdb350a36d624d0cc3f23589c6cab21fc/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/189ded48-fc9b-4109-828d-3c3cae8b5242/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ef2d819a1b2eb5b4a1d7f164b35868b83f8e06b0ba5504e6d3d0ebe1eab9457/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d14e549d-addd-40f6-b6af-12582e1a028b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82ef8e16-cb3d-4b88-9a4e-6970f544b650/volumes/kubernetes.io~projected/kube-api-access-rgzg5 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be44fdaf-13bc-4704-8c58-bcc0359528e8/volumes/kubernetes.io~projected/kube-api-access-6czjl DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03b436b1666a8957c1c82f9d9cdb3a2ae86c2aa3111fa67e71ab07930c229336/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9e7785c4-f87a-4969-9d54-1aafb3c2e641/volumes/kubernetes.io~projected/kube-api-access-dl4pq DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff4d1280f420614c96c12e23f03cdcbace6f9b6215c873e9e623625a3e94f9e3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2ef35a9e954227445b49998d78567e5aba76885a034f4cc2c45e79fef076d48/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ff325f-d384-407e-af38-2c876d9b686a/volumes/kubernetes.io~projected/kube-api-access-gl8qp DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4a482a0-025f-4bd6-8229-aa6971510e12/volumes/kubernetes.io~projected/kube-api-access-q9gqw DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f16e19f1-3da4-43db-b861-7146aa9dbede/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f16e19f1-3da4-43db-b861-7146aa9dbede/volumes/kubernetes.io~projected/kube-api-access-fbzkt DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/980055b40e485a207a302ffa51b3d739d54bff56247df5bc671c941e81aa85cb/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/671c112c9149b2bd8db675a1c244110b2d1173ed435e64b6a8ba1b33ac5d2ac7/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/228f642268f3ba6617452775130c97b2227f551091f8516a1d2f92cb013d9b2e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a780a4583a03dc51d5ff9a18f01711d63109e4f1e81debbcb77872106b61c1ff/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85cc6f7e-1889-40bb-a184-2138db4d29ba/volumes/kubernetes.io~projected/kube-api-access-45klt DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~projected/kube-api-access-ck8jm DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fcc1c87c926650c79308f45aed6ce979b3e78eeee4bc1763a49139452f6d65d/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26cdce1aa97ace90c9ebf3d9eb6f76b419a44958e235830ce0cf766201dbcbfe/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~projected/kube-api-access-r4cqc DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fea7ce1f-d9c4-40b4-a160-0f7788c416a6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84e7c05f-e63a-4829-ab28-c25bf61f9193/volumes/kubernetes.io~projected/kube-api-access-rh5ms DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~projected/kube-api-access-6mr9n DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e7244810ec14b9b66e58ce0518dce2254ad5a1b9b2f84b8de5a162c29e21f28/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52b4ca7a-3abb-4c17-a460-51f0b45cadca/volumes/kubernetes.io~projected/kube-api-access-t6z6h DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fccae350b6edf37351f93eaa658b8873d4e6f1a8e850b6a0b66ae8b19472e4d2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/add7fff3-f01d-4f58-893f-7cf024ae59e2/volumes/kubernetes.io~projected/kube-api-access-n4gvn DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be54cb41-84fc-4d14-a8c8-0435cf0954c9/volumes/kubernetes.io~projected/kube-api-access-xgfpv DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170/volumes/kubernetes.io~projected/kube-api-access-gcmxg DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24a6a6ac-45b7-4d97-9a4d-9f74db384832/volumes/kubernetes.io~projected/kube-api-access-gbn4h DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5c10760-26db-48ed-8fe8-92700a8fcf06/volumes/kubernetes.io~projected/kube-api-access-tr9cq DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b/volumes/kubernetes.io~projected/kube-api-access-4rjqh DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/947f945b-c140-4889-aaae-d332185913c7/volumes/kubernetes.io~projected/kube-api-access-z9zfz DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/add7fff3-f01d-4f58-893f-7cf024ae59e2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e4b6773962956f2d1372da9377a42524ec23f86a2bbe93dd6d5f66a74819134/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1804f9cc-5858-4cbb-a623-af98ee3234c2/volumes/kubernetes.io~projected/kube-api-access-cnfpf DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fdecbd0-dcd5-42f0-85ce-57b32e400992/volumes/kubernetes.io~projected/kube-api-access-z8qhm DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a889062-1b3c-442a-860d-64e9209a6acf/volumes/kubernetes.io~projected/kube-api-access-pr9qr DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2939006c-2a64-4684-968e-dc69c92c23ea/volumes/kubernetes.io~projected/kube-api-access-z6q8h DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/be54cb41-84fc-4d14-a8c8-0435cf0954c9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc1dba1-1e3a-4639-b2da-8277acf64efb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a9afd80-1122-46ad-8082-77f3b20738e9/volumes/kubernetes.io~projected/kube-api-access-xkgdq DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87c22c1e5b4990baab7426a46d15134e8cfc92138d38f5235d2a97efc1a0ca07/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bdb440db9bdcc2233377bd5611a99a3b72e276e2827592b71d3c6811fd2fff6/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/596ee397-e505-475e-aa1f-6d7096d8312b/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1804f9cc-5858-4cbb-a623-af98ee3234c2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a880088f3ef0a752e49f81be75567413272847828acf1ba597a5883a391bc8cd/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volumes/kubernetes.io~projected/kube-api-access-6cwfh DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e81006a-11bf-44aa-8561-1de0236016c8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~projected/kube-api-access-mql6m DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/189ded48-fc9b-4109-828d-3c3cae8b5242/volumes/kubernetes.io~projected/kube-api-access-s2kkk DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e2b0c34247ff094e999c42aedf01d2c017c032ed54df4e40ecc6e722f59cca7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4daae0fffba6f20b63e1dc34510996a2be78cf4bc6057ebb318e094c795d4ba/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03b436b1666a895 MacAddress:9e:80:a6:a3:a3:1c Speed:10000 Mtu:8901} {Name:0bdb440db9bdcc2 MacAddress:d2:76:74:a9:d8:ad Speed:10000 Mtu:8901} {Name:228f642268f3ba6 MacAddress:7e:12:ae:12:99:b9 Speed:10000 Mtu:8901} {Name:26cdce1aa97ace9 MacAddress:8a:2a:0d:f8:05:9d Speed:10000 Mtu:8901} {Name:5e7244810ec14b9 MacAddress:9e:eb:32:6b:c1:78 Speed:10000 Mtu:8901} {Name:5ef2d819a1b2eb5 MacAddress:ee:21:b1:7c:49:9c Speed:10000 Mtu:8901} {Name:87c22c1e5b4990b MacAddress:ea:54:74:13:72:b6 Speed:10000 Mtu:8901} {Name:980055b40e485a2 MacAddress:e6:ca:0c:97:3b:ed Speed:10000 Mtu:8901} {Name:a4daae0fffba6f2 MacAddress:46:a3:47:14:b9:2c Speed:10000 Mtu:8901} {Name:a780a4583a03dc5 MacAddress:06:5a:98:4a:8d:21 Speed:10000 Mtu:8901} {Name:a880088f3ef0a75 MacAddress:ba:b9:9c:69:dd:48 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e3:4e:4c:d1 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:03:bf:88:6b:96 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:e3:4e:4c:d1 Speed:0 Mtu:9001} {Name:ff4158ee425ebd1 MacAddress:d6:ca:d0:b9:f0:36 Speed:10000 Mtu:8901} {Name:ff4d1280f420614 MacAddress:d6:99:75:d1:03:b5 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:56:5f:57:84:4f:91 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:20:ae:4b:3d:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.129865 4966 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.129909 4966 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130201 4966 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130221 4966 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-174.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130447 4966 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130455 4966 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130466 4966 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130483 4966 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130503 4966 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130658 4966 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130763 4966 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130799 4966 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130817 4966 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130827 4966 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130836 4966 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:28:14.130841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.130849 4966 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:28:14.132079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.132059 4966 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:28:14.132079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.132079 4966 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:28:14.132778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.132756 4966 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:28:14.135423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135407 4966 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:28:14.135423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135422 4966 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:28:14.135534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135522 4966 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 19:28:14.135714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135702 4966 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:28:14.135744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135732 4966 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:28:14.135861 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135842 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135866 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135873 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135878 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135883 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135889 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135895 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135901 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135907 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135915 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135927 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:28:14.135941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135938 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:28:14.136249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135963 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:28:14.136249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.135973 4966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:28:14.139485 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.139470 4966 server.go:1263] "Started kubelet" Nov 02 19:28:14.139585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.139553 4966 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:28:14.139644 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.139570 4966 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:28:14.139833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.139647 4966 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:28:14.139970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.139951 4966 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:28:14.140011 ip-10-0-132-174 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:28:14.140846 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.140830 4966 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:28:14.146030 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.146012 4966 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:28:14.148081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.146035 4966 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:28:14.148081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.146087 4966 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:22:33 +0000 UTC" deadline="2028-06-15 15:13:55.382680561 +0000 UTC" Nov 02 19:28:14.148081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.146118 4966 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14179h45m41.236565404s" Nov 02 19:28:14.148081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.146139 4966 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:28:14.148081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.146148 4966 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:28:14.148081 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.146161 4966 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:28:14.148081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.146229 4966 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:28:14.148487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148200 4966 factory.go:55] Registering systemd factory Nov 02 19:28:14.148487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148219 4966 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:28:14.148596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148489 4966 factory.go:153] Registering CRI-O factory Nov 02 19:28:14.148596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148504 4966 factory.go:223] Registration of the crio container factory successfully Nov 02 19:28:14.148596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148562 4966 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:28:14.148596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148582 4966 factory.go:103] Registering Raw factory Nov 02 19:28:14.148596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148596 4966 manager.go:1199] Started watching for new ooms in manager Nov 02 19:28:14.148940 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.148923 4966 manager.go:319] Starting recovery of all containers Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149043 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c553-4232-4e4c-9770-494d8c427361" volumeName="kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149081 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e81006a-11bf-44aa-8561-1de0236016c8" volumeName="kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-script-lib" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149096 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e81006a-11bf-44aa-8561-1de0236016c8" volumeName="kubernetes.io/projected/9e81006a-11bf-44aa-8561-1de0236016c8-kube-api-access-6cwfh" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149110 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e81006a-11bf-44aa-8561-1de0236016c8" volumeName="kubernetes.io/secret/9e81006a-11bf-44aa-8561-1de0236016c8-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149124 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add7fff3-f01d-4f58-893f-7cf024ae59e2" volumeName="kubernetes.io/secret/add7fff3-f01d-4f58-893f-7cf024ae59e2-serving-cert" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149135 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb731f5a-7ec8-4440-b4b9-2ada95e8d40b" volumeName="kubernetes.io/projected/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b-kube-api-access-4rjqh" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149146 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1804f9cc-5858-4cbb-a623-af98ee3234c2" volumeName="kubernetes.io/projected/1804f9cc-5858-4cbb-a623-af98ee3234c2-kube-api-access-cnfpf" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149158 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1804f9cc-5858-4cbb-a623-af98ee3234c2" volumeName="kubernetes.io/secret/1804f9cc-5858-4cbb-a623-af98ee3234c2-serving-cert" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149172 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24a6a6ac-45b7-4d97-9a4d-9f74db384832" volumeName="kubernetes.io/projected/24a6a6ac-45b7-4d97-9a4d-9f74db384832-kube-api-access-gbn4h" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149184 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2939006c-2a64-4684-968e-dc69c92c23ea" volumeName="kubernetes.io/projected/2939006c-2a64-4684-968e-dc69c92c23ea-kube-api-access-z6q8h" seLinuxMountContext="" Nov 02 19:28:14.149190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149196 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85cc6f7e-1889-40bb-a184-2138db4d29ba" volumeName="kubernetes.io/projected/85cc6f7e-1889-40bb-a184-2138db4d29ba-kube-api-access-45klt" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149209 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5b533ad-a1e5-45ef-87ce-a0b2b37b7112" volumeName="kubernetes.io/projected/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-kube-api-access-wqv5g" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149220 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0656367c-7665-406c-af95-d47c276692aa" volumeName="kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149252 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c553-4232-4e4c-9770-494d8c427361" volumeName="kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149268 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e7785c4-f87a-4969-9d54-1aafb3c2e641" volumeName="kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cni-binary-copy" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149279 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be54cb41-84fc-4d14-a8c8-0435cf0954c9" volumeName="kubernetes.io/secret/be54cb41-84fc-4d14-a8c8-0435cf0954c9-serving-cert" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149292 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14e549d-addd-40f6-b6af-12582e1a028b" volumeName="kubernetes.io/configmap/d14e549d-addd-40f6-b6af-12582e1a028b-konnectivity-ca" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149334 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16e19f1-3da4-43db-b861-7146aa9dbede" volumeName="kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-service-ca-bundle" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149348 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82ef8e16-cb3d-4b88-9a4e-6970f544b650" volumeName="kubernetes.io/projected/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kube-api-access-rgzg5" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149360 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84e7c05f-e63a-4829-ab28-c25bf61f9193" volumeName="kubernetes.io/projected/84e7c05f-e63a-4829-ab28-c25bf61f9193-kube-api-access-rh5ms" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149372 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e81006a-11bf-44aa-8561-1de0236016c8" volumeName="kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-env-overrides" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149388 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0656367c-7665-406c-af95-d47c276692aa" volumeName="kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149400 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a889062-1b3c-442a-860d-64e9209a6acf" volumeName="kubernetes.io/projected/3a889062-1b3c-442a-860d-64e9209a6acf-kube-api-access-pr9qr" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149415 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc1dba1-1e3a-4639-b2da-8277acf64efb" volumeName="kubernetes.io/projected/3fc1dba1-1e3a-4639-b2da-8277acf64efb-kube-api-access-6mr9n" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149428 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c553-4232-4e4c-9770-494d8c427361" volumeName="kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149441 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c553-4232-4e4c-9770-494d8c427361" volumeName="kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149453 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170" volumeName="kubernetes.io/empty-dir/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-data-volume" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149464 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1804f9cc-5858-4cbb-a623-af98ee3234c2" volumeName="kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-trusted-ca" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149487 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b4ca7a-3abb-4c17-a460-51f0b45cadca" volumeName="kubernetes.io/empty-dir/52b4ca7a-3abb-4c17-a460-51f0b45cadca-tmp-dir" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149499 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add7fff3-f01d-4f58-893f-7cf024ae59e2" volumeName="kubernetes.io/projected/add7fff3-f01d-4f58-893f-7cf024ae59e2-kube-api-access-n4gvn" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149510 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16e19f1-3da4-43db-b861-7146aa9dbede" volumeName="kubernetes.io/secret/f16e19f1-3da4-43db-b861-7146aa9dbede-serving-cert" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149521 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0656367c-7665-406c-af95-d47c276692aa" volumeName="kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149532 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c553-4232-4e4c-9770-494d8c427361" volumeName="kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149546 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9afd80-1122-46ad-8082-77f3b20738e9" volumeName="kubernetes.io/projected/9a9afd80-1122-46ad-8082-77f3b20738e9-kube-api-access-xkgdq" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149559 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e7785c4-f87a-4969-9d54-1aafb3c2e641" volumeName="kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-daemon-config" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149571 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0656367c-7665-406c-af95-d47c276692aa" volumeName="kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149585 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1804f9cc-5858-4cbb-a623-af98ee3234c2" volumeName="kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-config" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149597 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdecbd0-dcd5-42f0-85ce-57b32e400992" volumeName="kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149608 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947f945b-c140-4889-aaae-d332185913c7" volumeName="kubernetes.io/configmap/947f945b-c140-4889-aaae-d332185913c7-config-volume" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149620 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596ee397-e505-475e-aa1f-6d7096d8312b" volumeName="kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-stats-auth" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149632 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170" volumeName="kubernetes.io/projected/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-api-access-gcmxg" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149643 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e81006a-11bf-44aa-8561-1de0236016c8" volumeName="kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-config" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149654 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16e19f1-3da4-43db-b861-7146aa9dbede" volumeName="kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149665 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0656367c-7665-406c-af95-d47c276692aa" volumeName="kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149676 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bf9d232-57b4-42b2-9919-b03b3170fe02" volumeName="kubernetes.io/configmap/7bf9d232-57b4-42b2-9919-b03b3170fe02-nginx-conf" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149687 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947f945b-c140-4889-aaae-d332185913c7" volumeName="kubernetes.io/empty-dir/947f945b-c140-4889-aaae-d332185913c7-tmp-dir" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149699 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be54cb41-84fc-4d14-a8c8-0435cf0954c9" volumeName="kubernetes.io/configmap/be54cb41-84fc-4d14-a8c8-0435cf0954c9-config" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149711 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be54cb41-84fc-4d14-a8c8-0435cf0954c9" volumeName="kubernetes.io/projected/be54cb41-84fc-4d14-a8c8-0435cf0954c9-kube-api-access-xgfpv" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149723 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0656367c-7665-406c-af95-d47c276692aa" volumeName="kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149735 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c553-4232-4e4c-9770-494d8c427361" volumeName="kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149754 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9afd80-1122-46ad-8082-77f3b20738e9" volumeName="kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149768 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be44fdaf-13bc-4704-8c58-bcc0359528e8" volumeName="kubernetes.io/projected/be44fdaf-13bc-4704-8c58-bcc0359528e8-kube-api-access-6czjl" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149780 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16e19f1-3da4-43db-b861-7146aa9dbede" volumeName="kubernetes.io/projected/f16e19f1-3da4-43db-b861-7146aa9dbede-kube-api-access-fbzkt" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149792 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189ded48-fc9b-4109-828d-3c3cae8b5242" volumeName="kubernetes.io/projected/189ded48-fc9b-4109-828d-3c3cae8b5242-kube-api-access-s2kkk" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149804 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ff325f-d384-407e-af38-2c876d9b686a" volumeName="kubernetes.io/projected/a3ff325f-d384-407e-af38-2c876d9b686a-kube-api-access-gl8qp" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149817 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fea7ce1f-d9c4-40b4-a160-0f7788c416a6" volumeName="kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149838 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0656367c-7665-406c-af95-d47c276692aa" volumeName="kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc" seLinuxMountContext="" Nov 02 19:28:14.149829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149850 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2939006c-2a64-4684-968e-dc69c92c23ea" volumeName="kubernetes.io/configmap/2939006c-2a64-4684-968e-dc69c92c23ea-telemetry-config" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149863 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b7c553-4232-4e4c-9770-494d8c427361" volumeName="kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149875 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947f945b-c140-4889-aaae-d332185913c7" volumeName="kubernetes.io/projected/947f945b-c140-4889-aaae-d332185913c7-kube-api-access-z9zfz" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149899 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4a482a0-025f-4bd6-8229-aa6971510e12" volumeName="kubernetes.io/configmap/c4a482a0-025f-4bd6-8229-aa6971510e12-iptables-alerter-script" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149910 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4a482a0-025f-4bd6-8229-aa6971510e12" volumeName="kubernetes.io/projected/c4a482a0-025f-4bd6-8229-aa6971510e12-kube-api-access-q9gqw" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149921 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d14e549d-addd-40f6-b6af-12582e1a028b" volumeName="kubernetes.io/secret/d14e549d-addd-40f6-b6af-12582e1a028b-agent-certs" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149932 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9afd80-1122-46ad-8082-77f3b20738e9" volumeName="kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149945 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add7fff3-f01d-4f58-893f-7cf024ae59e2" volumeName="kubernetes.io/configmap/add7fff3-f01d-4f58-893f-7cf024ae59e2-config" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149957 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be44fdaf-13bc-4704-8c58-bcc0359528e8" volumeName="kubernetes.io/configmap/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-config" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149968 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc1dba1-1e3a-4639-b2da-8277acf64efb" volumeName="kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-tuned" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149982 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9afd80-1122-46ad-8082-77f3b20738e9" volumeName="kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-binary-copy" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.149993 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e7785c4-f87a-4969-9d54-1aafb3c2e641" volumeName="kubernetes.io/projected/9e7785c4-f87a-4969-9d54-1aafb3c2e641-kube-api-access-dl4pq" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150006 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16e19f1-3da4-43db-b861-7146aa9dbede" volumeName="kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-snapshots" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150023 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16e19f1-3da4-43db-b861-7146aa9dbede" volumeName="kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-tmp" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150034 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc1dba1-1e3a-4639-b2da-8277acf64efb" volumeName="kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-tmp" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150046 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b4ca7a-3abb-4c17-a460-51f0b45cadca" volumeName="kubernetes.io/projected/52b4ca7a-3abb-4c17-a460-51f0b45cadca-kube-api-access-t6z6h" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150059 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596ee397-e505-475e-aa1f-6d7096d8312b" volumeName="kubernetes.io/projected/596ee397-e505-475e-aa1f-6d7096d8312b-kube-api-access-ck8jm" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150071 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="596ee397-e505-475e-aa1f-6d7096d8312b" volumeName="kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-default-certificate" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150082 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5b533ad-a1e5-45ef-87ce-a0b2b37b7112" volumeName="kubernetes.io/configmap/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-serviceca" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150096 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5c10760-26db-48ed-8fe8-92700a8fcf06" volumeName="kubernetes.io/projected/b5c10760-26db-48ed-8fe8-92700a8fcf06-kube-api-access-tr9cq" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150108 4966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="189ded48-fc9b-4109-828d-3c3cae8b5242" volumeName="kubernetes.io/secret/189ded48-fc9b-4109-828d-3c3cae8b5242-cert" seLinuxMountContext="" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150120 4966 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:28:14.152059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.150127 4966 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:28:14.165840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.165818 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/1.log" Nov 02 19:28:14.165954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.165878 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/1.log" Nov 02 19:28:14.173503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.173472 4966 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:28:14.175239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.175086 4966 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:28:14.175239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.175115 4966 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:28:14.175239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.175151 4966 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:28:14.175484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.175207 4966 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 19:28:14.211918 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.211883 4966 generic.go:356] "Generic (PLEG): container finished" podID="747e2de570be6f488466605f00369eb2" containerID="2b82d5eedd150e3e307eb7c46fc9c42269eb4a0132c4a141b7f9a8e4ad6a930c" exitCode=0 Nov 02 19:28:14.222066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.222039 4966 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="82c606c91fc7df34aba7d0b47483596abdafdcbb83949eba08375a05766cc2df" exitCode=0 Nov 02 19:28:14.222066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.222063 4966 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="a7878bd202102eb54b55adf4906d5df79012f3ab389038484076fa36277915f5" exitCode=0 Nov 02 19:28:14.222066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.222071 4966 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="c2306260e4aef8ae6f6ef0e4650e4c479e08447cd110361485f9ed647b964c79" exitCode=0 Nov 02 19:28:14.222320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.222082 4966 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="94f9fa9fb8831fee0e0ef24e0264bbab413a8ede5ba0aeec5738005aa1d6f450" exitCode=0 Nov 02 19:28:14.222320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.222091 4966 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="93f99487e1b0f52cff574e420a9c5931bb98a87638e478a8a810fca58a125072" exitCode=0 Nov 02 19:28:14.222320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.222098 4966 generic.go:356] "Generic (PLEG): container finished" podID="9a9afd80-1122-46ad-8082-77f3b20738e9" containerID="71b7414f79c8ff409863a766bac24d23a03f260fc48ce708ec81ae7d3919c1a0" exitCode=0 Nov 02 19:28:14.231540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.231514 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/1.log" Nov 02 19:28:14.231689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.231550 4966 generic.go:356] "Generic (PLEG): container finished" podID="1804f9cc-5858-4cbb-a623-af98ee3234c2" containerID="355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395" exitCode=255 Nov 02 19:28:14.239596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.239576 4966 manager.go:324] Recovery completed Nov 02 19:28:14.246217 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.246195 4966 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:28:14.255868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.255849 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:28:14.256965 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.256947 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:28:14.257032 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.256978 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:28:14.257032 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.256988 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:28:14.261180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.260896 4966 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:28:14.261180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.260921 4966 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:28:14.261180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.260944 4966 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:28:14.261180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261157 4966 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 19:28:14.261428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261171 4966 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 19:28:14.261428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261205 4966 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 19:28:14.261428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261213 4966 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 19:28:14.261428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261221 4966 policy_none.go:50] "Start" Nov 02 19:28:14.261428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261254 4966 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:28:14.261428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261271 4966 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:28:14.261428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261254 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:28:14.261629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261536 4966 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 19:28:14.261629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261548 4966 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 19:28:14.261629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.261555 4966 policy_none.go:44] "Start" Nov 02 19:28:14.262973 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.262952 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:28:14.263061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.262988 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:28:14.263061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.263002 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:28:14.266882 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.266857 4966 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:28:14.269372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269360 4966 manager.go:341] "Starting Device Plugin manager" Nov 02 19:28:14.269430 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.269384 4966 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:28:14.269430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269392 4966 server.go:87] "Starting device plugin registration server" Nov 02 19:28:14.269692 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269681 4966 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:28:14.269732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269693 4966 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:28:14.269776 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269756 4966 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:28:14.269878 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269838 4966 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:28:14.269878 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269876 4966 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:28:14.269996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269883 4966 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:28:14.269996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.269909 4966 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:28:14.269867576Z","UUID":"456c14d9-5952-4e5c-8e9a-7d8e4a1d047e","Handler":null,"Name":"","Endpoint":""} Nov 02 19:28:14.271279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.271257 4966 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:28:14.271391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.271321 4966 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:28:14.271718 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.271577 4966 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:28:14.271718 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.271614 4966 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 02 19:28:14.276130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.275586 4966 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal"] Nov 02 19:28:14.276235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.276221 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:28:14.276990 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.276973 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:28:14.277061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.277000 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:28:14.277061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.277012 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:28:14.277385 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.277372 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:28:14.277558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.277540 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:28:14.278083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278065 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:28:14.278169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278100 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:28:14.278169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278124 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:28:14.278169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278136 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:28:14.278169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278140 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:28:14.278169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278165 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:28:14.278409 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.278263 4966 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.278637 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278597 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"5b970eab0b7694596f8d763910ac4520ac56f6f0d9cb2d6412802669d3cc0127"} Nov 02 19:28:14.278721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278647 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:28:14.278721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278649 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerDied","Data":"2b82d5eedd150e3e307eb7c46fc9c42269eb4a0132c4a141b7f9a8e4ad6a930c"} Nov 02 19:28:14.278721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278673 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"32c827a75a38f305ceeb098b57e5285da019bed438e6df29776e2f0a056e8240"} Nov 02 19:28:14.278845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278728 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"bbbf4706e41de4207b1a770ebd3e743487e0db0d482f4edb6b9f8bcdd0ad3388"} Nov 02 19:28:14.278845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.278748 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"8ce1e7d03d1e62858b64de9e58ec948d18fb4554d02d13d8a450aa45c06c2f2a"} Nov 02 19:28:14.279386 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.279369 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:28:14.279445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.279402 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:28:14.279445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.279417 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:28:14.279563 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.279551 4966 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.326070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.326038 4966 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:28:14.326204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.326144 4966 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 19:28:14.326439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.326423 4966 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:28:14.326840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.326827 4966 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:28:14.346931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.346909 4966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.370146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.370083 4966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:28:14.371080 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.371065 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:28:14.371156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.371092 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:28:14.371156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.371105 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:28:14.371156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.371152 4966 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.372837 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.372819 4966 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:28:14.401653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.401630 4966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:28:14.401763 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.401679 4966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.401763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.401697 4966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.426268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.426236 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.426268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.426269 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.426433 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.426290 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.455149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.455122 4966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:28:14.455243 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:14.455171 4966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.473442 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.473415 4966 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.473554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.473489 4966 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.474090 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.474067 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeNotReady" Nov 02 19:28:14.474171 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.474106 4966 setters.go:546] "Node became not ready" node="ip-10-0-132-174.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T19:28:14Z","lastTransitionTime":"2026-11-02T19:28:14Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 19:28:14.493931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.493899 4966 nodeinfomanager.go:471] existing CSINode "ip-10-0-132-174.us-west-2.compute.internal" is owned by different node (oldNodeID="d75e73e5-1d63-4f75-827b-3471a7cf883b", newNodeID=""), cleaning up... Nov 02 19:28:14.526675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.526646 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.526675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.526679 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.526839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.526701 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.526839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.526758 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.526839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.526764 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.526839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.526809 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:14.737997 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.737969 4966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeReady" Nov 02 19:28:14.738145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:14.738072 4966 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 19:28:15.131413 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.131336 4966 apiserver.go:52] "Watching apiserver" Nov 02 19:28:15.237076 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.237051 4966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:15.237253 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.237207 4966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:15.275525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.275497 4966 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:28:15.276203 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.276175 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7","openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f","kube-system/global-pull-secret-syncer-2dw9d","openshift-ovn-kubernetes/ovnkube-node-8thp7","openshift-insights/insights-operator-7c4986d449-w8zlt","openshift-multus/network-metrics-daemon-7hnrd","openshift-console-operator/console-operator-7cd56b8dbc-jdcml","openshift-multus/multus-789jj","openshift-multus/multus-additional-cni-plugins-zs222","openshift-image-registry/image-registry-86c87c649b-jwznp","openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-txxgt","openshift-ingress-canary/ingress-canary-vpjl9","kube-system/kas-connection-checker-685895c7c9-52pkd","openshift-insights/insights-runtime-extractor-fb2w5","kube-system/kas-connection-checker-685895c7c9-k8shl","openshift-network-operator/iptables-alerter-jdlx5","openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n","openshift-image-registry/node-ca-j585w","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs","openshift-image-registry/image-registry-75f86d58f7-2pbzr","kube-system/konnectivity-agent-xfpnh","openshift-dns/node-resolver-s4b6j","openshift-ingress/router-default-9cf6bf74f-dljwv","openshift-cluster-node-tuning-operator/tuned-rglh2","openshift-dns/dns-default-xmrs7","openshift-network-console/networking-console-plugin-86c478bb88-fvrzl","openshift-network-diagnostics/network-check-target-wmg7k"] Nov 02 19:28:15.277838 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.277806 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.278434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.278413 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:15.278774 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.278748 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:15.280155 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.280137 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.280549 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.280534 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:15.281354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.281337 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:15.281827 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.281811 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.290558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.290536 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.291215 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.291200 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.297337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.297318 4966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:28:15.297415 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.297367 4966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:15.298816 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.298800 4966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:28:15.298864 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.298838 4966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 02 19:28:15.329425 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.329398 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 19:28:15.329531 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.329445 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 19:28:15.330291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.330273 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.331062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.331047 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.331339 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.331326 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 19:28:15.331576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.331564 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.331629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.331584 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 19:28:15.332054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.332033 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.333932 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.333912 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.334009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.333902 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.334099 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334083 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.334196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334156 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 19:28:15.334196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334182 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.334196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334174 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334202 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334211 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334218 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334228 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334235 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334240 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334248 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 19:28:15.334329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334250 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 19:28:15.334669 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334645 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 19:28:15.334765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334733 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.334824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334791 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.334860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334835 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.334957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334943 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 19:28:15.334994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.334956 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.335148 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.335137 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 19:28:15.336203 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.336189 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gsgfx\"" Nov 02 19:28:15.336508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.336389 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 19:28:15.336508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.336388 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2skqm\"" Nov 02 19:28:15.336508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.336415 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.343699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.343681 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.343699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.343697 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 19:28:15.343845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.343709 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.343845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.343728 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.343845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.343777 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 19:28:15.343845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.343840 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:28:15.345027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.345011 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 19:28:15.345102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.345044 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.345102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.345057 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.348889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.348875 4966 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:28:15.377235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.377216 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.377510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.377495 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 19:28:15.378419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.378404 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.378509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.378497 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 19:28:15.378552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.378512 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.378672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.378657 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.380365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.380345 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 19:28:15.380636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.380621 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 19:28:15.429420 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429336 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:28:15.429420 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429381 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.429633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429510 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:28:15.429633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429522 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:28:15.429633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429577 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 19:28:15.429633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429591 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4cqc\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.429633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429630 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429601 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429627 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-kubernetes\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429669 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429680 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mr9n\" (UniqueName: \"kubernetes.io/projected/3fc1dba1-1e3a-4639-b2da-8277acf64efb-kube-api-access-6mr9n\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429707 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q8h\" (UniqueName: \"kubernetes.io/projected/2939006c-2a64-4684-968e-dc69c92c23ea-kube-api-access-z6q8h\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429737 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d14e549d-addd-40f6-b6af-12582e1a028b-konnectivity-ca\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429770 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-k8s-cni-cncf-io\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.429875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429795 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e81006a-11bf-44aa-8561-1de0236016c8-ovn-node-metrics-cert\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429900 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429982 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.429982 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430015 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430032 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430041 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430035 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-daemon-config\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430083 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wxdrr\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430092 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430113 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430134 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ktd87\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430154 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430167 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwfh\" (UniqueName: \"kubernetes.io/projected/9e81006a-11bf-44aa-8561-1de0236016c8-kube-api-access-6cwfh\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430178 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430226 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sp8wk\"" Nov 02 19:28:15.430251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430221 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430272 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430239 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430319 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-modprobe-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430346 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430377 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430522 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430540 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lnn7q\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430672 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430674 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430704 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430710 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430719 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430743 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430770 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-c8nwr\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430679 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t4nhx\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430865 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430887 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k9tfs\"" Nov 02 19:28:15.430913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430895 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430967 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.430968 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431087 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-multus-certs\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431150 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431168 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431203 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-var-lib-kubelet\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431157 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.431574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431270 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bb8t\"" Nov 02 19:28:15.432086 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431719 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbvgr\"" Nov 02 19:28:15.432172 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432014 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-tmp\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.432172 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432022 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dc6gd\"" Nov 02 19:28:15.432424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432045 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-tmp\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.432503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432463 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.432503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432482 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.432503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432493 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-netd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.432690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432539 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9gqw\" (UniqueName: \"kubernetes.io/projected/c4a482a0-025f-4bd6-8229-aa6971510e12-kube-api-access-q9gqw\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:28:15.432690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432488 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.432819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432716 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.432819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432729 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nnqx8\"" Nov 02 19:28:15.432819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432762 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.433003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432913 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-netns\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.433003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.432954 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4pq\" (UniqueName: \"kubernetes.io/projected/9e7785c4-f87a-4969-9d54-1aafb3c2e641-kube-api-access-dl4pq\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.433128 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433011 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-dbus\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.433128 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433052 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mql6m\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.433261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433090 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cnibin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.433261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433193 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-conf-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.433261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433226 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-var-lib-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.433261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433255 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:28:15.433516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433293 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:28:15.433516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433257 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m9rmj\"" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433661 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-serviceca\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433793 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.431205 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434079 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.433257 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-serviceca\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434161 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434201 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-ovn\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434234 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434291 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434239 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434542 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fzxnx\"" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434569 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-registration-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434662 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434713 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-service-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434767 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434844 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434949 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-service-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.434993 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-run\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435026 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n2mjf\"" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435075 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435078 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-etc-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435115 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435179 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435199 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9t85s\"" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435214 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/947f945b-c140-4889-aaae-d332185913c7-config-volume\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.435377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435272 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435430 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh5ms\" (UniqueName: \"kubernetes.io/projected/84e7c05f-e63a-4829-ab28-c25bf61f9193-kube-api-access-rh5ms\") pod \"kas-connection-checker-685895c7c9-k8shl\" (UID: \"84e7c05f-e63a-4829-ab28-c25bf61f9193\") " pod="kube-system/kas-connection-checker-685895c7c9-k8shl" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435472 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-netns\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435507 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqv5g\" (UniqueName: \"kubernetes.io/projected/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-kube-api-access-wqv5g\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435567 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bg56f\"" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435602 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/947f945b-c140-4889-aaae-d332185913c7-config-volume\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435630 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/add7fff3-f01d-4f58-893f-7cf024ae59e2-config\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435713 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-slash\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435750 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-stats-auth\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435826 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9cq\" (UniqueName: \"kubernetes.io/projected/b5c10760-26db-48ed-8fe8-92700a8fcf06-kube-api-access-tr9cq\") pod \"volume-data-source-validator-58c9f5f46d-snj5g\" (UID: \"b5c10760-26db-48ed-8fe8-92700a8fcf06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435846 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435901 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/add7fff3-f01d-4f58-893f-7cf024ae59e2-config\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.435930 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9zfz\" (UniqueName: \"kubernetes.io/projected/947f945b-c140-4889-aaae-d332185913c7-kube-api-access-z9zfz\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436146 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-stats-auth\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436197 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysconfig\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436344 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-bin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436430 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-systemd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436516 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-lib-modules\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436567 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-tuned\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436603 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzkt\" (UniqueName: \"kubernetes.io/projected/f16e19f1-3da4-43db-b861-7146aa9dbede-kube-api-access-fbzkt\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436650 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-multus\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.436716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436718 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-host\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436747 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-sys\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436782 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-device-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436818 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be54cb41-84fc-4d14-a8c8-0435cf0954c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436852 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1804f9cc-5858-4cbb-a623-af98ee3234c2-serving-cert\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436882 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgdq\" (UniqueName: \"kubernetes.io/projected/9a9afd80-1122-46ad-8082-77f3b20738e9-kube-api-access-xkgdq\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436916 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-socket-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.436977 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437026 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437068 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-config\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437114 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437153 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-config\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437225 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjqh\" (UniqueName: \"kubernetes.io/projected/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b-kube-api-access-4rjqh\") pod \"network-check-source-6f78d59f99-mz8m7\" (UID: \"cb731f5a-7ec8-4440-b4b9-2ada95e8d40b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437350 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52b4ca7a-3abb-4c17-a460-51f0b45cadca-hosts-file\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437403 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a482a0-025f-4bd6-8229-aa6971510e12-host-slash\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437465 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437540 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437604 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437634 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-hostroot\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437670 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-kubelet\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.437737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437672 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437761 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bf9d232-57b4-42b2-9919-b03b3170fe02-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437812 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-systemd\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437847 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-host\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437888 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-config\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437973 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-tuned\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.437988 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-cnibin\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438026 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-default-certificate\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438151 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-original-pull-secret\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438195 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16e19f1-3da4-43db-b861-7146aa9dbede-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438192 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-socket-dir-parent\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438192 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bf9d232-57b4-42b2-9919-b03b3170fe02-nginx-conf\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438367 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-systemd-units\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438449 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-script-lib\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438464 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-default-certificate\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438479 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1804f9cc-5858-4cbb-a623-af98ee3234c2-config\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438510 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438611 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1804f9cc-5858-4cbb-a623-af98ee3234c2-serving-cert\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.438664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438643 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8jm\" (UniqueName: \"kubernetes.io/projected/596ee397-e505-475e-aa1f-6d7096d8312b-kube-api-access-ck8jm\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438726 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-ovnkube-script-lib\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438784 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-data-volume\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438838 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-data-volume\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438858 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-system-cni-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.438895 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-env-overrides\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439009 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439075 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45klt\" (UniqueName: \"kubernetes.io/projected/85cc6f7e-1889-40bb-a184-2138db4d29ba-kube-api-access-45klt\") pod \"kas-connection-checker-685895c7c9-52pkd\" (UID: \"85cc6f7e-1889-40bb-a184-2138db4d29ba\") " pod="kube-system/kas-connection-checker-685895c7c9-52pkd" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439136 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/add7fff3-f01d-4f58-893f-7cf024ae59e2-serving-cert\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439159 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e81006a-11bf-44aa-8561-1de0236016c8-env-overrides\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439274 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439292 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439330 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/add7fff3-f01d-4f58-893f-7cf024ae59e2-serving-cert\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439347 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-kubelet\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439360 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.439415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439396 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6z6h\" (UniqueName: \"kubernetes.io/projected/52b4ca7a-3abb-4c17-a460-51f0b45cadca-kube-api-access-t6z6h\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:28:15.439876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439434 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbn4h\" (UniqueName: \"kubernetes.io/projected/24a6a6ac-45b7-4d97-9a4d-9f74db384832-kube-api-access-gbn4h\") pod \"migrator-66df5d496b-fzv6f\" (UID: \"24a6a6ac-45b7-4d97-9a4d-9f74db384832\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" Nov 02 19:28:15.439876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439527 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-tmp\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.439876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439551 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-bin\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.439876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439631 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-tmp\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.439876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439636 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.439876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439707 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfpv\" (UniqueName: \"kubernetes.io/projected/be54cb41-84fc-4d14-a8c8-0435cf0954c9-kube-api-access-xgfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:15.439876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439735 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-system-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439866 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439905 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9qr\" (UniqueName: \"kubernetes.io/projected/3a889062-1b3c-442a-860d-64e9209a6acf-kube-api-access-pr9qr\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.439962 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440002 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kkk\" (UniqueName: \"kubernetes.io/projected/189ded48-fc9b-4109-828d-3c3cae8b5242-kube-api-access-s2kkk\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440038 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440097 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-sys-fs\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440038 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.440134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440095 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e81006a-11bf-44aa-8561-1de0236016c8-ovn-node-metrics-cert\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440158 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440162 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be54cb41-84fc-4d14-a8c8-0435cf0954c9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440208 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d14e549d-addd-40f6-b6af-12582e1a028b-konnectivity-ca\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440260 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcmxg\" (UniqueName: \"kubernetes.io/projected/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-api-access-gcmxg\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440352 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-log-socket\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440387 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfpf\" (UniqueName: \"kubernetes.io/projected/1804f9cc-5858-4cbb-a623-af98ee3234c2-kube-api-access-cnfpf\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440401 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be54cb41-84fc-4d14-a8c8-0435cf0954c9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:15.440461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440413 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-conf\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.440728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440479 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzg5\" (UniqueName: \"kubernetes.io/projected/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kube-api-access-rgzg5\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.440728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440520 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-daemon-config\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.440728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440588 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b4ca7a-3abb-4c17-a460-51f0b45cadca-tmp-dir\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:28:15.440728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440667 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8qp\" (UniqueName: \"kubernetes.io/projected/a3ff325f-d384-407e-af38-2c876d9b686a-kube-api-access-gl8qp\") pod \"kas-connection-checker-685895c7c9-txxgt\" (UID: \"a3ff325f-d384-407e-af38-2c876d9b686a\") " pod="kube-system/kas-connection-checker-685895c7c9-txxgt" Nov 02 19:28:15.440728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440696 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b4ca7a-3abb-4c17-a460-51f0b45cadca-tmp-dir\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:28:15.440880 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440738 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.440880 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440799 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-node-log\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.440880 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440817 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:15.440981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440888 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-crio-socket\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.440981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.440931 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4a482a0-025f-4bd6-8229-aa6971510e12-iptables-alerter-script\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:28:15.441038 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.441009 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-kubelet-config\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.441080 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.441060 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-etc-kubernetes\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.441164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.441105 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189ded48-fc9b-4109-828d-3c3cae8b5242-cert\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:15.441323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.441286 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4a482a0-025f-4bd6-8229-aa6971510e12-iptables-alerter-script\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:28:15.441412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.441335 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/189ded48-fc9b-4109-828d-3c3cae8b5242-cert\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:15.448406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448388 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.448480 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448421 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-snapshots\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.448480 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448440 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d14e549d-addd-40f6-b6af-12582e1a028b-agent-certs\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:28:15.448480 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448439 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be54cb41-84fc-4d14-a8c8-0435cf0954c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:15.448480 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448461 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.448480 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448481 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448496 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2939006c-2a64-4684-968e-dc69c92c23ea-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448518 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f16e19f1-3da4-43db-b861-7146aa9dbede-serving-cert\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448524 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f16e19f1-3da4-43db-b861-7146aa9dbede-snapshots\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448560 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448613 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448667 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gvn\" (UniqueName: \"kubernetes.io/projected/add7fff3-f01d-4f58-893f-7cf024ae59e2-kube-api-access-n4gvn\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448670 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f16e19f1-3da4-43db-b861-7146aa9dbede-serving-cert\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.448716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448679 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d14e549d-addd-40f6-b6af-12582e1a028b-agent-certs\") pod \"konnectivity-agent-xfpnh\" (UID: \"d14e549d-addd-40f6-b6af-12582e1a028b\") " pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448753 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448787 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czjl\" (UniqueName: \"kubernetes.io/projected/be44fdaf-13bc-4704-8c58-bcc0359528e8-kube-api-access-6czjl\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448852 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448885 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2939006c-2a64-4684-968e-dc69c92c23ea-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448909 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cni-binary-copy\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448951 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947f945b-c140-4889-aaae-d332185913c7-tmp-dir\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.448980 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.449039 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947f945b-c140-4889-aaae-d332185913c7-tmp-dir\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.449080 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cni-binary-copy\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.449240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.449230 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9afd80-1122-46ad-8082-77f3b20738e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.481284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.481261 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mr9n\" (UniqueName: \"kubernetes.io/projected/3fc1dba1-1e3a-4639-b2da-8277acf64efb-kube-api-access-6mr9n\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.483809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.483784 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9gqw\" (UniqueName: \"kubernetes.io/projected/c4a482a0-025f-4bd6-8229-aa6971510e12-kube-api-access-q9gqw\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:28:15.483886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.483828 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q8h\" (UniqueName: \"kubernetes.io/projected/2939006c-2a64-4684-968e-dc69c92c23ea-kube-api-access-z6q8h\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:15.483886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.483834 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwfh\" (UniqueName: \"kubernetes.io/projected/9e81006a-11bf-44aa-8561-1de0236016c8-kube-api-access-6cwfh\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.483957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.483882 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.483957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.483934 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4cqc\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.484909 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.484891 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4pq\" (UniqueName: \"kubernetes.io/projected/9e7785c4-f87a-4969-9d54-1aafb3c2e641-kube-api-access-dl4pq\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.485060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.485045 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mql6m\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.487194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.487147 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8qhm\" (UniqueName: \"kubernetes.io/projected/1fdecbd0-dcd5-42f0-85ce-57b32e400992-kube-api-access-z8qhm\") pod \"network-check-target-wmg7k\" (UID: \"1fdecbd0-dcd5-42f0-85ce-57b32e400992\") " pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:15.487450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.487431 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqv5g\" (UniqueName: \"kubernetes.io/projected/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-kube-api-access-wqv5g\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:28:15.487537 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.487500 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh5ms\" (UniqueName: \"kubernetes.io/projected/84e7c05f-e63a-4829-ab28-c25bf61f9193-kube-api-access-rh5ms\") pod \"kas-connection-checker-685895c7c9-k8shl\" (UID: \"84e7c05f-e63a-4829-ab28-c25bf61f9193\") " pod="kube-system/kas-connection-checker-685895c7c9-k8shl" Nov 02 19:28:15.487669 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.487651 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9zfz\" (UniqueName: \"kubernetes.io/projected/947f945b-c140-4889-aaae-d332185913c7-kube-api-access-z9zfz\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.488358 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.488340 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9cq\" (UniqueName: \"kubernetes.io/projected/b5c10760-26db-48ed-8fe8-92700a8fcf06-kube-api-access-tr9cq\") pod \"volume-data-source-validator-58c9f5f46d-snj5g\" (UID: \"b5c10760-26db-48ed-8fe8-92700a8fcf06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-snj5g" Nov 02 19:28:15.490402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.490371 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45klt\" (UniqueName: \"kubernetes.io/projected/85cc6f7e-1889-40bb-a184-2138db4d29ba-kube-api-access-45klt\") pod \"kas-connection-checker-685895c7c9-52pkd\" (UID: \"85cc6f7e-1889-40bb-a184-2138db4d29ba\") " pod="kube-system/kas-connection-checker-685895c7c9-52pkd" Nov 02 19:28:15.490545 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.490517 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjqh\" (UniqueName: \"kubernetes.io/projected/cb731f5a-7ec8-4440-b4b9-2ada95e8d40b-kube-api-access-4rjqh\") pod \"network-check-source-6f78d59f99-mz8m7\" (UID: \"cb731f5a-7ec8-4440-b4b9-2ada95e8d40b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mz8m7" Nov 02 19:28:15.490725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.490707 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzkt\" (UniqueName: \"kubernetes.io/projected/f16e19f1-3da4-43db-b861-7146aa9dbede-kube-api-access-fbzkt\") pod \"insights-operator-7c4986d449-w8zlt\" (UID: \"f16e19f1-3da4-43db-b861-7146aa9dbede\") " pod="openshift-insights/insights-operator-7c4986d449-w8zlt" Nov 02 19:28:15.490793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.490775 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8jm\" (UniqueName: \"kubernetes.io/projected/596ee397-e505-475e-aa1f-6d7096d8312b-kube-api-access-ck8jm\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.490998 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.490975 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgdq\" (UniqueName: \"kubernetes.io/projected/9a9afd80-1122-46ad-8082-77f3b20738e9-kube-api-access-xkgdq\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.493065 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.493041 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcmxg\" (UniqueName: \"kubernetes.io/projected/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-api-access-gcmxg\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.493322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.493286 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbn4h\" (UniqueName: \"kubernetes.io/projected/24a6a6ac-45b7-4d97-9a4d-9f74db384832-kube-api-access-gbn4h\") pod \"migrator-66df5d496b-fzv6f\" (UID: \"24a6a6ac-45b7-4d97-9a4d-9f74db384832\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fzv6f" Nov 02 19:28:15.493594 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.493568 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzg5\" (UniqueName: \"kubernetes.io/projected/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kube-api-access-rgzg5\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.493666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.493648 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kkk\" (UniqueName: \"kubernetes.io/projected/189ded48-fc9b-4109-828d-3c3cae8b5242-kube-api-access-s2kkk\") pod \"ingress-canary-vpjl9\" (UID: \"189ded48-fc9b-4109-828d-3c3cae8b5242\") " pod="openshift-ingress-canary/ingress-canary-vpjl9" Nov 02 19:28:15.493770 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.493740 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9qr\" (UniqueName: \"kubernetes.io/projected/3a889062-1b3c-442a-860d-64e9209a6acf-kube-api-access-pr9qr\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:15.494081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.494064 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfpv\" (UniqueName: \"kubernetes.io/projected/be54cb41-84fc-4d14-a8c8-0435cf0954c9-kube-api-access-xgfpv\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs\" (UID: \"be54cb41-84fc-4d14-a8c8-0435cf0954c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" Nov 02 19:28:15.494171 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.494144 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6z6h\" (UniqueName: \"kubernetes.io/projected/52b4ca7a-3abb-4c17-a460-51f0b45cadca-kube-api-access-t6z6h\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:28:15.494227 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.494200 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8qp\" (UniqueName: \"kubernetes.io/projected/a3ff325f-d384-407e-af38-2c876d9b686a-kube-api-access-gl8qp\") pod \"kas-connection-checker-685895c7c9-txxgt\" (UID: \"a3ff325f-d384-407e-af38-2c876d9b686a\") " pod="kube-system/kas-connection-checker-685895c7c9-txxgt" Nov 02 19:28:15.494554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.494524 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.494835 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.494819 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfpf\" (UniqueName: \"kubernetes.io/projected/1804f9cc-5858-4cbb-a623-af98ee3234c2-kube-api-access-cnfpf\") pod \"console-operator-7cd56b8dbc-jdcml\" (UID: \"1804f9cc-5858-4cbb-a623-af98ee3234c2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:15.499599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.499580 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czjl\" (UniqueName: \"kubernetes.io/projected/be44fdaf-13bc-4704-8c58-bcc0359528e8-kube-api-access-6czjl\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:15.499697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.499681 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gvn\" (UniqueName: \"kubernetes.io/projected/add7fff3-f01d-4f58-893f-7cf024ae59e2-kube-api-access-n4gvn\") pod \"service-ca-operator-544d749c8c-d9l6n\" (UID: \"add7fff3-f01d-4f58-893f-7cf024ae59e2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" Nov 02 19:28:15.549440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549418 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:15.549534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549445 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-sys-fs\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.549534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549462 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-log-socket\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.549534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549476 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-conf\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.549534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549501 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-node-log\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549535 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-log-socket\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549524 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549548 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-sys-fs\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549566 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-node-log\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549580 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-crio-socket\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549606 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-kubelet-config\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549629 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-etc-kubernetes\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.549606 4966 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549647 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-crio-socket\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549638 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-conf\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549664 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549668 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-kubelet-config\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549692 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-etc-kubernetes\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.549716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549695 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-kubernetes\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549732 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-k8s-cni-cncf-io\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.549621 4966 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549759 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.549768 4966 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.549799 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.049771391 +0000 UTC m=+1.998427226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549807 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-k8s-cni-cncf-io\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549799 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-kubernetes\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.549827 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.04980922 +0000 UTC m=+1.998465043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549847 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549878 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549875 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysctl-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.549902 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.549911 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.049900325 +0000 UTC m=+1.998556141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : secret "metrics-daemon-secret" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549924 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549930 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-modprobe-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549965 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.549992 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550006 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.049993789 +0000 UTC m=+1.998649621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550044 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-multus-certs\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550058 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-modprobe-d\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550068 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-multus-certs\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550070 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-var-lib-kubelet\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550097 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550055 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550112 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-netd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550131 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-var-lib-kubelet\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550146 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-netd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550049 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550175 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550174 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-netns\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550201 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.050190698 +0000 UTC m=+1.998846518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550214 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-run-netns\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550227 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-dbus\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550244 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cnibin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.550235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550260 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-conf-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550277 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-var-lib-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550323 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-conf-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550321 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-ovn\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550349 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fea7ce1f-d9c4-40b4-a160-0f7788c416a6-dbus\") pod \"global-pull-secret-syncer-2dw9d\" (UID: \"fea7ce1f-d9c4-40b4-a160-0f7788c416a6\") " pod="kube-system/global-pull-secret-syncer-2dw9d" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550363 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550384 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-ovn\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550402 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-cnibin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550391 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-registration-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550366 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-var-lib-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550414 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550440 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550460 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-registration-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550468 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550497 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-run\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550508 4966 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550515 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550522 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-etc-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550538 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.050529673 +0000 UTC m=+1.999185524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550555 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550555 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-etc-openvswitch\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550558 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-run\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550572 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550589 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-netns\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550604 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-slash\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550623 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysconfig\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550650 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-bin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550662 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-slash\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550665 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-run-netns\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550676 4966 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550695 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-systemd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550721 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550734 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550737 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-run-systemd\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550742 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.050726688 +0000 UTC m=+1.999382508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550688 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-bin\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550720 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-lib-modules\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550767 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-sysconfig\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.550788 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.050775572 +0000 UTC m=+1.999431409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550804 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-multus\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550838 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-cni-multus\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550808 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-lib-modules\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550873 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-host\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550842 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5b533ad-a1e5-45ef-87ce-a0b2b37b7112-host\") pod \"node-ca-j585w\" (UID: \"a5b533ad-a1e5-45ef-87ce-a0b2b37b7112\") " pod="openshift-image-registry/node-ca-j585w" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550896 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-sys\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550911 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-device-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550927 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-socket-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550943 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550958 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550976 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52b4ca7a-3abb-4c17-a460-51f0b45cadca-hosts-file\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550983 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-device-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.550991 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a482a0-025f-4bd6-8229-aa6971510e12-host-slash\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551000 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-sys\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551014 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551042 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-hostroot\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551044 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52b4ca7a-3abb-4c17-a460-51f0b45cadca-hosts-file\") pod \"node-resolver-s4b6j\" (UID: \"52b4ca7a-3abb-4c17-a460-51f0b45cadca\") " pod="openshift-dns/node-resolver-s4b6j" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551045 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82ef8e16-cb3d-4b88-9a4e-6970f544b650-socket-dir\") pod \"aws-ebs-csi-driver-node-fjzx7\" (UID: \"82ef8e16-cb3d-4b88-9a4e-6970f544b650\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjzx7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551083 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-hostroot\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.551088 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.551097 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.051050233 +0000 UTC m=+1.999706109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.551104 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551116 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a482a0-025f-4bd6-8229-aa6971510e12-host-slash\") pod \"iptables-alerter-jdlx5\" (UID: \"c4a482a0-025f-4bd6-8229-aa6971510e12\") " pod="openshift-network-operator/iptables-alerter-jdlx5" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:15.551147 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:16.051136496 +0000 UTC m=+1.999792319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551167 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-kubelet\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551193 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-systemd\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551217 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-host\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551240 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-cnibin\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551265 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-socket-dir-parent\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551281 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-etc-systemd\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551284 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-kubelet\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551312 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-systemd-units\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551337 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-systemd-units\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551337 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fc1dba1-1e3a-4639-b2da-8277acf64efb-host\") pod \"tuned-rglh2\" (UID: \"3fc1dba1-1e3a-4639-b2da-8277acf64efb\") " pod="openshift-cluster-node-tuning-operator/tuned-rglh2" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551345 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551317 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-cnibin\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551379 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-multus-socket-dir-parent\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551389 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-system-cni-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551433 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-kubelet\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551439 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551432 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9afd80-1122-46ad-8082-77f3b20738e9-system-cni-dir\") pod \"multus-additional-cni-plugins-zs222\" (UID: \"9a9afd80-1122-46ad-8082-77f3b20738e9\") " pod="openshift-multus/multus-additional-cni-plugins-zs222" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551462 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-host-var-lib-kubelet\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551475 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-bin\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551506 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-system-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551508 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551533 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551557 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-cni-bin\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.551524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551579 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e7785c4-f87a-4969-9d54-1aafb3c2e641-system-cni-dir\") pod \"multus-789jj\" (UID: \"9e7785c4-f87a-4969-9d54-1aafb3c2e641\") " pod="openshift-multus/multus-789jj" Nov 02 19:28:15.553831 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.551585 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e81006a-11bf-44aa-8561-1de0236016c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8thp7\" (UID: \"9e81006a-11bf-44aa-8561-1de0236016c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:15.579677 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.579650 4966 scope.go:122] "RemoveContainer" containerID="355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395" Nov 02 19:28:15.927954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:15.927929 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:16.056183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056094 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:16.056183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056143 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:16.056183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056166 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056192 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056220 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056256 4966 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056319 4966 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056260 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056340 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056354 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056333521 +0000 UTC m=+3.004989353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056342 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056394 4966 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056339 4966 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056431 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056416543 +0000 UTC m=+3.005072364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056449 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056442021 +0000 UTC m=+3.005097838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056474 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056459468 +0000 UTC m=+3.005115285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:16.056484 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056492 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.0564867 +0000 UTC m=+3.005142517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : secret "metrics-daemon-secret" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056508 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.05650255 +0000 UTC m=+3.005158367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056539 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056561 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056599 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.056620 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056649 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056664 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056653 4966 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056700 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056682993 +0000 UTC m=+3.005338827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056705 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056734 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056723 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056713324 +0000 UTC m=+3.005369147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056780 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056770259 +0000 UTC m=+3.005426079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:16.056963 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.056795 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:17.056787431 +0000 UTC m=+3.005443248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:16.241841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.241811 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/2.log" Nov 02 19:28:16.242265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.242250 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/1.log" Nov 02 19:28:16.242326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.242284 4966 generic.go:356] "Generic (PLEG): container finished" podID="1804f9cc-5858-4cbb-a623-af98ee3234c2" containerID="86581d7c8256398d172c114ec739b95f4d87229020a8413883136ba19d72e3a9" exitCode=255 Nov 02 19:28:16.242360 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.242342 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" event={"ID":"1804f9cc-5858-4cbb-a623-af98ee3234c2","Type":"ContainerDied","Data":"86581d7c8256398d172c114ec739b95f4d87229020a8413883136ba19d72e3a9"} Nov 02 19:28:16.242410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.242396 4966 scope.go:122] "RemoveContainer" containerID="355321360d38c0bc135b7e6cbda2939b0de8ca54737887cf3864236672687395" Nov 02 19:28:16.243254 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:16.243228 4966 scope.go:122] "RemoveContainer" containerID="86581d7c8256398d172c114ec739b95f4d87229020a8413883136ba19d72e3a9" Nov 02 19:28:16.243451 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:16.243429 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:17.070060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070030 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:17.070060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070067 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070098 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070119 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070138 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070172 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070192 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070202 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070245 4966 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070192 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070262 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070274 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070284 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070273 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070254503 +0000 UTC m=+5.018910332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070294 4966 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070337 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070323049 +0000 UTC m=+5.018978867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:17.070333 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070318 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070350 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070344395 +0000 UTC m=+5.019000211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070364 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070358347 +0000 UTC m=+5.019014163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070375 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070370554 +0000 UTC m=+5.019026371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070386 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.07038108 +0000 UTC m=+5.019036898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070397 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070391924 +0000 UTC m=+5.019047740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070432 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070454 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.070477 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070549 4966 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070556 4966 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070579 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070570623 +0000 UTC m=+5.019226444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070601 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070589801 +0000 UTC m=+5.019245618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070561 4966 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 19:28:17.070775 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:17.070632 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:28:19.070624621 +0000 UTC m=+5.019280438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : secret "metrics-daemon-secret" not found Nov 02 19:28:17.108366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.108333 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:17.248004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.247975 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/2.log" Nov 02 19:28:17.248401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:17.248173 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:18.485599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:18.485560 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:18.498013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:18.497980 4966 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:18.498291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:18.498278 4966 scope.go:122] "RemoveContainer" containerID="86581d7c8256398d172c114ec739b95f4d87229020a8413883136ba19d72e3a9" Nov 02 19:28:18.498482 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:18.498464 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:19.088226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088190 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:19.088226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088231 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088259 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088289 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088356 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088379 4966 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088386 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088413 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088431 4966 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088443 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls podName:947f945b-c140-4889-aaae-d332185913c7 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088425097 +0000 UTC m=+9.037080920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls") pod "dns-default-xmrs7" (UID: "947f945b-c140-4889-aaae-d332185913c7") : secret "dns-default-metrics-tls" not found Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088439 4966 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088381 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088487 4966 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088493 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:19.088508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088491 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls podName:be44fdaf-13bc-4704-8c58-bcc0359528e8 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088472972 +0000 UTC m=+9.037128793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-j82gn" (UID: "be44fdaf-13bc-4704-8c58-bcc0359528e8") : secret "samples-operator-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088538 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088381 4966 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088578 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088569812 +0000 UTC m=+9.037225629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : secret "metrics-daemon-secret" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088574 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.088617 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088624 4966 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088512 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86c87c649b-jwznp: secret "image-registry-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088638 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088622344 +0000 UTC m=+9.037278161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : configmap references non-existent config key: service-ca.crt Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088658 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088649556 +0000 UTC m=+9.037305377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088637 4966 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75f86d58f7-2pbzr: secret "image-registry-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088675 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088666966 +0000 UTC m=+9.037322788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088661 4966 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088691 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs podName:596ee397-e505-475e-aa1f-6d7096d8312b nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088682011 +0000 UTC m=+9.037337834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs") pod "router-default-9cf6bf74f-dljwv" (UID: "596ee397-e505-475e-aa1f-6d7096d8312b") : secret "router-metrics-certs-default" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088705 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls podName:0656367c-7665-406c-af95-d47c276692aa nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088698001 +0000 UTC m=+9.037353823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls") pod "image-registry-86c87c649b-jwznp" (UID: "0656367c-7665-406c-af95-d47c276692aa") : secret "image-registry-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088724 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls podName:74b7c553-4232-4e4c-9770-494d8c427361 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088716304 +0000 UTC m=+9.037372137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls") pod "image-registry-75f86d58f7-2pbzr" (UID: "74b7c553-4232-4e4c-9770-494d8c427361") : secret "image-registry-tls" not found Nov 02 19:28:19.088971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.088744 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:23.088735955 +0000 UTC m=+9.037391780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:19.701886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.701857 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4b6j_52b4ca7a-3abb-4c17-a460-51f0b45cadca/dns-node-resolver/0.log" Nov 02 19:28:19.719864 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.719826 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:19.720328 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:19.720310 4966 scope.go:122] "RemoveContainer" containerID="86581d7c8256398d172c114ec739b95f4d87229020a8413883136ba19d72e3a9" Nov 02 19:28:19.720558 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:19.720536 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:20.903523 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:20.903496 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j585w_a5b533ad-a1e5-45ef-87ce-a0b2b37b7112/node-ca/0.log" Nov 02 19:28:21.501788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:21.501763 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vpjl9_189ded48-fc9b-4109-828d-3c3cae8b5242/serve-healthcheck-canary/0.log" Nov 02 19:28:21.703058 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:21.703029 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fzv6f_24a6a6ac-45b7-4d97-9a4d-9f74db384832/migrator/0.log" Nov 02 19:28:22.105309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:22.105276 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs_be54cb41-84fc-4d14-a8c8-0435cf0954c9/kube-storage-version-migrator-operator/0.log" Nov 02 19:28:23.126902 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.126868 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.126911 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.126941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.126963 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127027 4966 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127058 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127089 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls podName:2939006c-2a64-4684-968e-dc69c92c23ea nodeName:}" failed. No retries permitted until 2026-11-02 19:28:31.127074874 +0000 UTC m=+17.075730720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ffh8p" (UID: "2939006c-2a64-4684-968e-dc69c92c23ea") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127030 4966 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127135 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127170 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127199 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls podName:9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:31.127177642 +0000 UTC m=+17.075833470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fb2w5" (UID: "9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170") : secret "insights-runtime-extractor-tls" not found Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127340 4966 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:28:23.127410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127377 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:23.128074 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127421 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:23.128074 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127459 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:23.128074 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127539 4966 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 19:28:23.128074 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127593 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs podName:3a889062-1b3c-442a-860d-64e9209a6acf nodeName:}" failed. No retries permitted until 2026-11-02 19:28:31.127576458 +0000 UTC m=+17.076232277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs") pod "network-metrics-daemon-7hnrd" (UID: "3a889062-1b3c-442a-860d-64e9209a6acf") : secret "metrics-daemon-secret" not found Nov 02 19:28:23.128074 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127688 4966 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:28:23.128074 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:23.127747 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert podName:7bf9d232-57b4-42b2-9919-b03b3170fe02 nodeName:}" failed. No retries permitted until 2026-11-02 19:28:31.127732853 +0000 UTC m=+17.076388687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-fvrzl" (UID: "7bf9d232-57b4-42b2-9919-b03b3170fe02") : secret "networking-console-plugin-cert" not found Nov 02 19:28:23.128074 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.127905 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/596ee397-e505-475e-aa1f-6d7096d8312b-service-ca-bundle\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:23.129769 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.129736 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/947f945b-c140-4889-aaae-d332185913c7-metrics-tls\") pod \"dns-default-xmrs7\" (UID: \"947f945b-c140-4889-aaae-d332185913c7\") " pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:23.129905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.129737 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"image-registry-86c87c649b-jwznp\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:23.130070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.130052 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"image-registry-75f86d58f7-2pbzr\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:23.130178 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.130155 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/596ee397-e505-475e-aa1f-6d7096d8312b-metrics-certs\") pod \"router-default-9cf6bf74f-dljwv\" (UID: \"596ee397-e505-475e-aa1f-6d7096d8312b\") " pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:23.130217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.130162 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be44fdaf-13bc-4704-8c58-bcc0359528e8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-j82gn\" (UID: \"be44fdaf-13bc-4704-8c58-bcc0359528e8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:23.379267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.379179 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:23.379267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.379203 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" Nov 02 19:28:23.381309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.381284 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:23.383080 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.383057 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:23.391584 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.391562 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:23.753638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.753606 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn"] Nov 02 19:28:23.804337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.804309 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75f86d58f7-2pbzr"] Nov 02 19:28:23.827903 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.827886 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:28:23.840712 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:23.840691 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b7c553_4232_4e4c_9770_494d8c427361.slice/crio-417b199bc7c73b365436821ebcca7bb3ecd5f20113e09f0a6fa24dca02af1b20 WatchSource:0}: Error finding container 417b199bc7c73b365436821ebcca7bb3ecd5f20113e09f0a6fa24dca02af1b20: Status 404 returned error can't find the container with id 417b199bc7c73b365436821ebcca7bb3ecd5f20113e09f0a6fa24dca02af1b20 Nov 02 19:28:23.841409 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:23.841339 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0656367c_7665_406c_af95_d47c276692aa.slice/crio-654cc6b7422214ef3a6e1cae1c49dc1b62817c1964bf6b6ebef085b03e94bb7f WatchSource:0}: Error finding container 654cc6b7422214ef3a6e1cae1c49dc1b62817c1964bf6b6ebef085b03e94bb7f: Status 404 returned error can't find the container with id 654cc6b7422214ef3a6e1cae1c49dc1b62817c1964bf6b6ebef085b03e94bb7f Nov 02 19:28:23.850580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.850561 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86c87c649b-jwznp"] Nov 02 19:28:23.851003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.850982 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmrs7"] Nov 02 19:28:23.851971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:23.851951 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9cf6bf74f-dljwv"] Nov 02 19:28:23.856012 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:23.855990 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947f945b_c140_4889_aaae_d332185913c7.slice/crio-83b3f836fa5b8919bba55685043bd7b0d642d7c61afa0da631230a768a3a0d58 WatchSource:0}: Error finding container 83b3f836fa5b8919bba55685043bd7b0d642d7c61afa0da631230a768a3a0d58: Status 404 returned error can't find the container with id 83b3f836fa5b8919bba55685043bd7b0d642d7c61afa0da631230a768a3a0d58 Nov 02 19:28:23.857451 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:23.857429 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596ee397_e505_475e_aa1f_6d7096d8312b.slice/crio-ad9b42ed3c7f96178fc34a19093c237a990e770c36f413a530855295eb0b6c33 WatchSource:0}: Error finding container ad9b42ed3c7f96178fc34a19093c237a990e770c36f413a530855295eb0b6c33: Status 404 returned error can't find the container with id ad9b42ed3c7f96178fc34a19093c237a990e770c36f413a530855295eb0b6c33 Nov 02 19:28:24.273736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.273700 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmrs7" event={"ID":"947f945b-c140-4889-aaae-d332185913c7","Type":"ContainerStarted","Data":"83b3f836fa5b8919bba55685043bd7b0d642d7c61afa0da631230a768a3a0d58"} Nov 02 19:28:24.275886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.275427 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" event={"ID":"596ee397-e505-475e-aa1f-6d7096d8312b","Type":"ContainerStarted","Data":"a5944a501980ae031ecdf1ccb8c723c85ece2ae1bd89dd4656217749f6ebfdc7"} Nov 02 19:28:24.275886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.275484 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" event={"ID":"596ee397-e505-475e-aa1f-6d7096d8312b","Type":"ContainerStarted","Data":"ad9b42ed3c7f96178fc34a19093c237a990e770c36f413a530855295eb0b6c33"} Nov 02 19:28:24.277117 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.277088 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" event={"ID":"0656367c-7665-406c-af95-d47c276692aa","Type":"ContainerStarted","Data":"d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef"} Nov 02 19:28:24.277202 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.277137 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" event={"ID":"0656367c-7665-406c-af95-d47c276692aa","Type":"ContainerStarted","Data":"654cc6b7422214ef3a6e1cae1c49dc1b62817c1964bf6b6ebef085b03e94bb7f"} Nov 02 19:28:24.278926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.278860 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" event={"ID":"74b7c553-4232-4e4c-9770-494d8c427361","Type":"ContainerStarted","Data":"47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a"} Nov 02 19:28:24.278926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.278902 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" event={"ID":"74b7c553-4232-4e4c-9770-494d8c427361","Type":"ContainerStarted","Data":"417b199bc7c73b365436821ebcca7bb3ecd5f20113e09f0a6fa24dca02af1b20"} Nov 02 19:28:24.280289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.280263 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" event={"ID":"be44fdaf-13bc-4704-8c58-bcc0359528e8","Type":"ContainerStarted","Data":"01050278fd30a4c3a4f007fcb7c827eafeb04cbeaaaf8d6dd7b4905d46766475"} Nov 02 19:28:24.380116 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.380087 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:24.382443 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.382423 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:24.618609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.618514 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:24.619622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:24.619601 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-9cf6bf74f-dljwv" Nov 02 19:28:25.284529 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:25.284441 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:25.284529 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:25.284448 4966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:28:25.449630 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:25.449591 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xfpnh" Nov 02 19:28:26.290277 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:26.290240 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmrs7" event={"ID":"947f945b-c140-4889-aaae-d332185913c7","Type":"ContainerStarted","Data":"80a7d1dd83bae7bb3faca9327fed6f084a7c2eba7031e49fa465d0bde3c58519"} Nov 02 19:28:26.290673 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:26.290317 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmrs7" event={"ID":"947f945b-c140-4889-aaae-d332185913c7","Type":"ContainerStarted","Data":"f62b626b9025125060b1a53691476a57addf67430057291bc82170516278804e"} Nov 02 19:28:27.297241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:27.297190 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" event={"ID":"be44fdaf-13bc-4704-8c58-bcc0359528e8","Type":"ContainerStarted","Data":"61e16b9f2d9ec9d862a9f78258e6cfab28aa13ef16b4380f4daedcd18f3add0f"} Nov 02 19:28:27.297673 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:27.297308 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-j82gn" event={"ID":"be44fdaf-13bc-4704-8c58-bcc0359528e8","Type":"ContainerStarted","Data":"08c83fbbdaf340c3302080d99dae6b3383765d32856a8d967e5ffc489aa36570"} Nov 02 19:28:27.297673 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:27.297340 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:28.176099 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.176066 4966 scope.go:122] "RemoveContainer" containerID="86581d7c8256398d172c114ec739b95f4d87229020a8413883136ba19d72e3a9" Nov 02 19:28:28.302604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.302584 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:28:28.303029 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.303013 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/2.log" Nov 02 19:28:28.303083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.303044 4966 generic.go:356] "Generic (PLEG): container finished" podID="1804f9cc-5858-4cbb-a623-af98ee3234c2" containerID="ecf47310d77776b3c8fb1d5f4aff92892da5a959611bfc6f32f45080d1e09b1b" exitCode=255 Nov 02 19:28:28.303138 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.303117 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" event={"ID":"1804f9cc-5858-4cbb-a623-af98ee3234c2","Type":"ContainerDied","Data":"ecf47310d77776b3c8fb1d5f4aff92892da5a959611bfc6f32f45080d1e09b1b"} Nov 02 19:28:28.303180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.303169 4966 scope.go:122] "RemoveContainer" containerID="86581d7c8256398d172c114ec739b95f4d87229020a8413883136ba19d72e3a9" Nov 02 19:28:28.303674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.303655 4966 scope.go:122] "RemoveContainer" containerID="ecf47310d77776b3c8fb1d5f4aff92892da5a959611bfc6f32f45080d1e09b1b" Nov 02 19:28:28.303838 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:28.303818 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:28.498809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:28.498783 4966 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:29.307361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:29.307333 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:28:29.307832 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:29.307727 4966 scope.go:122] "RemoveContainer" containerID="ecf47310d77776b3c8fb1d5f4aff92892da5a959611bfc6f32f45080d1e09b1b" Nov 02 19:28:29.307974 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:29.307952 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:29.720528 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:29.720494 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:30.310776 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:30.310748 4966 scope.go:122] "RemoveContainer" containerID="ecf47310d77776b3c8fb1d5f4aff92892da5a959611bfc6f32f45080d1e09b1b" Nov 02 19:28:30.311180 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:30.310933 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdcml_openshift-console-operator(1804f9cc-5858-4cbb-a623-af98ee3234c2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" Nov 02 19:28:31.202475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.202442 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:31.202664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.202480 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:31.202664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.202522 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:31.202664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.202555 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:31.204883 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.204857 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fb2w5\" (UID: \"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170\") " pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:31.204969 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.204894 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a889062-1b3c-442a-860d-64e9209a6acf-metrics-certs\") pod \"network-metrics-daemon-7hnrd\" (UID: \"3a889062-1b3c-442a-860d-64e9209a6acf\") " pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:31.204969 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.204923 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bf9d232-57b4-42b2-9919-b03b3170fe02-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-fvrzl\" (UID: \"7bf9d232-57b4-42b2-9919-b03b3170fe02\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:31.205068 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.205051 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2939006c-2a64-4684-968e-dc69c92c23ea-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ffh8p\" (UID: \"2939006c-2a64-4684-968e-dc69c92c23ea\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:31.233479 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.233452 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8thp7" Nov 02 19:28:31.480033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.480003 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" Nov 02 19:28:31.481115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.481095 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" Nov 02 19:28:31.482161 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.482146 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7hnrd" Nov 02 19:28:31.492466 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.492448 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fb2w5" Nov 02 19:28:31.911163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.911094 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-fvrzl"] Nov 02 19:28:31.914208 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:31.914181 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf9d232_57b4_42b2_9919_b03b3170fe02.slice/crio-03ff6107aa61caaa80929757e0a66f8f696360a0c65269e9e526a49098dbfa12 WatchSource:0}: Error finding container 03ff6107aa61caaa80929757e0a66f8f696360a0c65269e9e526a49098dbfa12: Status 404 returned error can't find the container with id 03ff6107aa61caaa80929757e0a66f8f696360a0c65269e9e526a49098dbfa12 Nov 02 19:28:31.957622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.957593 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p"] Nov 02 19:28:31.958394 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:31.958372 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7hnrd"] Nov 02 19:28:31.961132 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:31.961108 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a889062_1b3c_442a_860d_64e9209a6acf.slice/crio-8804cef922473d5a0bc0a8bf6563b6d7724b0400dec00e49bfd183b298407d60 WatchSource:0}: Error finding container 8804cef922473d5a0bc0a8bf6563b6d7724b0400dec00e49bfd183b298407d60: Status 404 returned error can't find the container with id 8804cef922473d5a0bc0a8bf6563b6d7724b0400dec00e49bfd183b298407d60 Nov 02 19:28:31.961884 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:31.961773 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2939006c_2a64_4684_968e_dc69c92c23ea.slice/crio-3aedbf954f9ddda0f5d15412762ddd4cc5e9960a7241ca883c9037b58ea0c191 WatchSource:0}: Error finding container 3aedbf954f9ddda0f5d15412762ddd4cc5e9960a7241ca883c9037b58ea0c191: Status 404 returned error can't find the container with id 3aedbf954f9ddda0f5d15412762ddd4cc5e9960a7241ca883c9037b58ea0c191 Nov 02 19:28:31.963857 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:31.963831 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca9cca5_9ce8_4824_bf86_9b3bbd8fc170.slice/crio-56cf6b03b21f8b492eda6749df9851c59efc20128ef1da985f66abf34ccdb163 WatchSource:0}: Error finding container 56cf6b03b21f8b492eda6749df9851c59efc20128ef1da985f66abf34ccdb163: Status 404 returned error can't find the container with id 56cf6b03b21f8b492eda6749df9851c59efc20128ef1da985f66abf34ccdb163 Nov 02 19:28:32.002946 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.002926 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fb2w5"] Nov 02 19:28:32.081774 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.081736 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:32.318222 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.318187 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hnrd" event={"ID":"3a889062-1b3c-442a-860d-64e9209a6acf","Type":"ContainerStarted","Data":"8804cef922473d5a0bc0a8bf6563b6d7724b0400dec00e49bfd183b298407d60"} Nov 02 19:28:32.319329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.319310 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" event={"ID":"7bf9d232-57b4-42b2-9919-b03b3170fe02","Type":"ContainerStarted","Data":"03ff6107aa61caaa80929757e0a66f8f696360a0c65269e9e526a49098dbfa12"} Nov 02 19:28:32.320479 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.320451 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" event={"ID":"2939006c-2a64-4684-968e-dc69c92c23ea","Type":"ContainerStarted","Data":"3aedbf954f9ddda0f5d15412762ddd4cc5e9960a7241ca883c9037b58ea0c191"} Nov 02 19:28:32.321814 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.321795 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fb2w5" event={"ID":"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170","Type":"ContainerStarted","Data":"70fc00993a0819062e561cba49a6498db6f0e4224c7216b08328014100121c90"} Nov 02 19:28:32.321889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.321832 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fb2w5" event={"ID":"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170","Type":"ContainerStarted","Data":"56cf6b03b21f8b492eda6749df9851c59efc20128ef1da985f66abf34ccdb163"} Nov 02 19:28:32.462538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:32.462497 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:34.333270 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:34.332470 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hnrd" event={"ID":"3a889062-1b3c-442a-860d-64e9209a6acf","Type":"ContainerStarted","Data":"5a42826ae329cc90af192a18fecbd032d90ca0eefb6cca20900d0aedb2c7e055"} Nov 02 19:28:34.333270 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:34.332529 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7hnrd" event={"ID":"3a889062-1b3c-442a-860d-64e9209a6acf","Type":"ContainerStarted","Data":"a6198751746fe2ddde83b02cc9ea8b4eec6232a07e1cab377b71d5648e6430cc"} Nov 02 19:28:34.336174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:34.336144 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-fvrzl" event={"ID":"7bf9d232-57b4-42b2-9919-b03b3170fe02","Type":"ContainerStarted","Data":"b5f31ede83108a72bd27d681b702f7dbbcdc4d4184e1f855e8ed4bbdae00ad0a"} Nov 02 19:28:34.339873 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:34.339851 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fb2w5" event={"ID":"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170","Type":"ContainerStarted","Data":"c887cbbc9741785dc88875ecd2c11e3c9bece81f113d58eb4d31f5d33eaa8424"} Nov 02 19:28:34.815937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:34.815903 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f86d58f7-2pbzr"] Nov 02 19:28:35.345018 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:35.344978 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ffh8p" event={"ID":"2939006c-2a64-4684-968e-dc69c92c23ea","Type":"ContainerStarted","Data":"a7d2aaaa14df532b6c56cfd920f40386cf66e19fb2261b827e0369df7bdc3039"} Nov 02 19:28:35.347758 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:35.347730 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fb2w5" event={"ID":"9ca9cca5-9ce8-4824-bf86-9b3bbd8fc170","Type":"ContainerStarted","Data":"444be534837d75b7a2b46491a05ae932fe4fdd5c6a3dbfec59b213a0b73c0131"} Nov 02 19:28:37.305506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:37.305465 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xmrs7" Nov 02 19:28:41.500354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:41.500286 4966 patch_prober.go:29] interesting pod/image-registry-86c87c649b-jwznp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 19:28:41.500737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:41.500376 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" podUID="0656367c-7665-406c-af95-d47c276692aa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:28:42.086397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:42.086360 4966 patch_prober.go:29] interesting pod/image-registry-86c87c649b-jwznp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 19:28:42.086562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:42.086416 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" podUID="0656367c-7665-406c-af95-d47c276692aa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:28:44.750281 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:28:44.750241 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:28:44.821325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:44.821284 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:28:47.550501 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.550463 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t5ppq"] Nov 02 19:28:47.586623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.586592 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.696847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.696809 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njtg7\"" Nov 02 19:28:47.697023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.696902 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:28:47.697023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.696965 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:28:47.742196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742157 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-textfile\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742199 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742448 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742337 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jp69\" (UniqueName: \"kubernetes.io/projected/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-kube-api-access-9jp69\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742448 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742384 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742448 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742434 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-tls\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742594 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742454 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-metrics-client-ca\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742651 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742623 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-wtmp\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742774 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-root\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742824 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-sys\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.742976 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.742958 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:28:47.743040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.743029 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:28:47.843676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843593 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-root\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843630 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-sys\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843650 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-textfile\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843671 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843709 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jp69\" (UniqueName: \"kubernetes.io/projected/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-kube-api-access-9jp69\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843725 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-sys\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843725 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-root\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843739 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843814 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-tls\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843841 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-metrics-client-ca\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.843999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.843920 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-wtmp\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.844262 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.844100 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-textfile\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.844262 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.844135 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-wtmp\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.844365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.844336 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.844438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.844421 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-metrics-client-ca\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.846239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.846218 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-tls\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.846333 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.846280 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.897525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.897491 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jp69\" (UniqueName: \"kubernetes.io/projected/62c8a1d4-ef6e-45ea-b0ee-3354a556b978-kube-api-access-9jp69\") pod \"node-exporter-t5ppq\" (UID: \"62c8a1d4-ef6e-45ea-b0ee-3354a556b978\") " pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.924869 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:47.924838 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5ppq" Nov 02 19:28:47.960782 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:28:47.960748 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c8a1d4_ef6e_45ea_b0ee_3354a556b978.slice/crio-c7f6e5996ecf7f4a7009635e5ca5494e63324e0af53d602c98d6bc06e5df8eea WatchSource:0}: Error finding container c7f6e5996ecf7f4a7009635e5ca5494e63324e0af53d602c98d6bc06e5df8eea: Status 404 returned error can't find the container with id c7f6e5996ecf7f4a7009635e5ca5494e63324e0af53d602c98d6bc06e5df8eea Nov 02 19:28:48.394757 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:48.394722 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ppq" event={"ID":"62c8a1d4-ef6e-45ea-b0ee-3354a556b978","Type":"ContainerStarted","Data":"c7f6e5996ecf7f4a7009635e5ca5494e63324e0af53d602c98d6bc06e5df8eea"} Nov 02 19:28:48.488108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:48.488076 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wmg7k" Nov 02 19:28:50.175716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.175648 4966 scope.go:122] "RemoveContainer" containerID="ecf47310d77776b3c8fb1d5f4aff92892da5a959611bfc6f32f45080d1e09b1b" Nov 02 19:28:50.404514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.404481 4966 generic.go:356] "Generic (PLEG): container finished" podID="62c8a1d4-ef6e-45ea-b0ee-3354a556b978" containerID="9e570c0eb46f51fba9dcfb36e2ef14f261432aba1faa843accc780f7e0b3a5da" exitCode=0 Nov 02 19:28:50.404686 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.404570 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ppq" event={"ID":"62c8a1d4-ef6e-45ea-b0ee-3354a556b978","Type":"ContainerDied","Data":"9e570c0eb46f51fba9dcfb36e2ef14f261432aba1faa843accc780f7e0b3a5da"} Nov 02 19:28:50.406549 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.406529 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:28:50.406664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.406569 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" event={"ID":"1804f9cc-5858-4cbb-a623-af98ee3234c2","Type":"ContainerStarted","Data":"93af8476d264a3990606f787adfd4dc292fe93d6d20062f26f29a9a9e03ecbe5"} Nov 02 19:28:50.406847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.406825 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:50.408203 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.408166 4966 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jdcml container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Nov 02 19:28:50.408263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:50.408219 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" podUID="1804f9cc-5858-4cbb-a623-af98ee3234c2" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Nov 02 19:28:51.413357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:51.412659 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ppq" event={"ID":"62c8a1d4-ef6e-45ea-b0ee-3354a556b978","Type":"ContainerStarted","Data":"0bec01b6e0546d3ebd8c34150d042255bd51cc037a333905260f1d35fe22b09a"} Nov 02 19:28:51.413357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:51.412772 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5ppq" event={"ID":"62c8a1d4-ef6e-45ea-b0ee-3354a556b978","Type":"ContainerStarted","Data":"99587a4184e54438f1908f067536680c0256ceccb82fe1a865cdedf3bb9369cc"} Nov 02 19:28:51.420396 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:51.420368 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdcml" Nov 02 19:28:51.565643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:51.565581 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t5ppq" podStartSLOduration=2.679785468 podStartE2EDuration="4.565567035s" totalImagesPullingTime="1.802149616s" totalInitContainerRuntime="83.631951ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:28:51.564522506 +0000 UTC m=+37.513178348" watchObservedRunningTime="2026-11-02 19:28:51.565567035 +0000 UTC m=+37.514222871" Nov 02 19:28:52.085903 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:52.085871 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:28:57.862854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:57.862776 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86c87c649b-jwznp"] Nov 02 19:28:59.837991 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:28:59.837952 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" podUID="74b7c553-4232-4e4c-9770-494d8c427361" containerName="registry" containerID="cri-o://47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a" gracePeriod=30 Nov 02 19:29:00.072808 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.072778 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:29:00.144110 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144026 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.144110 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144063 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.144110 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144086 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.144110 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144103 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.144430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144126 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.144430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144150 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.146204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144623 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.146204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.144745 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\") pod \"74b7c553-4232-4e4c-9770-494d8c427361\" (UID: \"74b7c553-4232-4e4c-9770-494d8c427361\") " Nov 02 19:29:00.146204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.145713 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:29:00.146591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.146561 4966 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-registry-certificates\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.148389 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.148349 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:29:00.150626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.150448 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "kube-api-access-mql6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:29:00.152329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.150751 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:29:00.152329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.151437 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:29:00.152905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.152879 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:29:00.154293 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.154268 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:29:00.156072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.156047 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted" pod "74b7c553-4232-4e4c-9770-494d8c427361" (UID: "74b7c553-4232-4e4c-9770-494d8c427361"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:29:00.247358 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.247319 4966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74b7c553-4232-4e4c-9770-494d8c427361-trusted-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.247358 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.247351 4966 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-installation-pull-secrets\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.247358 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.247362 4966 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74b7c553-4232-4e4c-9770-494d8c427361-ca-trust-extracted\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.247634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.247372 4966 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74b7c553-4232-4e4c-9770-494d8c427361-image-registry-private-configuration\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.247634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.247381 4966 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-registry-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.247634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.247389 4966 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-bound-sa-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.247634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.247398 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mql6m\" (UniqueName: \"kubernetes.io/projected/74b7c553-4232-4e4c-9770-494d8c427361-kube-api-access-mql6m\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:00.446227 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.446139 4966 generic.go:356] "Generic (PLEG): container finished" podID="74b7c553-4232-4e4c-9770-494d8c427361" containerID="47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a" exitCode=0 Nov 02 19:29:00.446227 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.446207 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" Nov 02 19:29:00.446440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.446221 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" event={"ID":"74b7c553-4232-4e4c-9770-494d8c427361","Type":"ContainerDied","Data":"47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a"} Nov 02 19:29:00.446440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.446273 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75f86d58f7-2pbzr" event={"ID":"74b7c553-4232-4e4c-9770-494d8c427361","Type":"ContainerDied","Data":"417b199bc7c73b365436821ebcca7bb3ecd5f20113e09f0a6fa24dca02af1b20"} Nov 02 19:29:00.446440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.446293 4966 scope.go:122] "RemoveContainer" containerID="47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a" Nov 02 19:29:00.454744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.454726 4966 scope.go:122] "RemoveContainer" containerID="47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a" Nov 02 19:29:00.455052 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:29:00.455031 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a\": container with ID starting with 47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a not found: ID does not exist" containerID="47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a" Nov 02 19:29:00.455132 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.455061 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a"} err="failed to get container status \"47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a\": rpc error: code = NotFound desc = could not find container \"47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a\": container with ID starting with 47f4eb8f97c49a44905d2749edebb8268f0ba9a4d163868508d3b910e2b13d5a not found: ID does not exist" Nov 02 19:29:00.604513 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.604479 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75f86d58f7-2pbzr"] Nov 02 19:29:00.614368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:00.614342 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75f86d58f7-2pbzr"] Nov 02 19:29:02.180798 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:02.180761 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b7c553-4232-4e4c-9770-494d8c427361" path="/var/lib/kubelet/pods/74b7c553-4232-4e4c-9770-494d8c427361/volumes" Nov 02 19:29:14.750546 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:29:14.750512 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:29:22.882699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:22.882637 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" podUID="0656367c-7665-406c-af95-d47c276692aa" containerName="registry" containerID="cri-o://d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef" gracePeriod=30 Nov 02 19:29:23.134828 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.134769 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:29:23.241072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241040 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241233 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241083 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241233 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241207 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241250 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241270 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241289 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241333 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241367 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\") pod \"0656367c-7665-406c-af95-d47c276692aa\" (UID: \"0656367c-7665-406c-af95-d47c276692aa\") " Nov 02 19:29:23.241566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241552 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:29:23.241671 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.241631 4966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-trusted-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.242093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.242053 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:29:23.244188 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.244143 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:29:23.244328 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.244274 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:29:23.244489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.244464 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:29:23.244613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.244485 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "kube-api-access-r4cqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:29:23.244613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.244529 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:29:23.248724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.248635 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted" pod "0656367c-7665-406c-af95-d47c276692aa" (UID: "0656367c-7665-406c-af95-d47c276692aa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:29:23.342538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.342504 4966 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0656367c-7665-406c-af95-d47c276692aa-ca-trust-extracted\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.342538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.342535 4966 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-image-registry-private-configuration\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.342538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.342545 4966 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0656367c-7665-406c-af95-d47c276692aa-installation-pull-secrets\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.342775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.342555 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4cqc\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-kube-api-access-r4cqc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.342775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.342565 4966 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0656367c-7665-406c-af95-d47c276692aa-registry-certificates\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.342775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.342574 4966 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-bound-sa-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.342775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.342583 4966 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0656367c-7665-406c-af95-d47c276692aa-registry-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:29:23.534920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.534887 4966 generic.go:356] "Generic (PLEG): container finished" podID="0656367c-7665-406c-af95-d47c276692aa" containerID="d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef" exitCode=0 Nov 02 19:29:23.535094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.534932 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" Nov 02 19:29:23.535094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.534964 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" event={"ID":"0656367c-7665-406c-af95-d47c276692aa","Type":"ContainerDied","Data":"d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef"} Nov 02 19:29:23.535094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.535016 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c87c649b-jwznp" event={"ID":"0656367c-7665-406c-af95-d47c276692aa","Type":"ContainerDied","Data":"654cc6b7422214ef3a6e1cae1c49dc1b62817c1964bf6b6ebef085b03e94bb7f"} Nov 02 19:29:23.535094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.535038 4966 scope.go:122] "RemoveContainer" containerID="d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef" Nov 02 19:29:23.543892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.543876 4966 scope.go:122] "RemoveContainer" containerID="d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef" Nov 02 19:29:23.544159 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:29:23.544141 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef\": container with ID starting with d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef not found: ID does not exist" containerID="d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef" Nov 02 19:29:23.544201 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.544167 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef"} err="failed to get container status \"d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef\": rpc error: code = NotFound desc = could not find container \"d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef\": container with ID starting with d3932b94220d0701ec6d1c51dcabaeadf5929f2da6b827e38214c600b577f3ef not found: ID does not exist" Nov 02 19:29:23.686281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.686244 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86c87c649b-jwznp"] Nov 02 19:29:23.689514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:23.689490 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86c87c649b-jwznp"] Nov 02 19:29:24.180431 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:24.180398 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0656367c-7665-406c-af95-d47c276692aa" path="/var/lib/kubelet/pods/0656367c-7665-406c-af95-d47c276692aa/volumes" Nov 02 19:29:34.575168 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:34.575078 4966 generic.go:356] "Generic (PLEG): container finished" podID="be54cb41-84fc-4d14-a8c8-0435cf0954c9" containerID="fe085febe6eba893d886b8efd1192ba33f322b6637a532d1a4d39a319fdc1360" exitCode=0 Nov 02 19:29:34.575168 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:34.575154 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" event={"ID":"be54cb41-84fc-4d14-a8c8-0435cf0954c9","Type":"ContainerDied","Data":"fe085febe6eba893d886b8efd1192ba33f322b6637a532d1a4d39a319fdc1360"} Nov 02 19:29:34.575716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:34.575555 4966 scope.go:122] "RemoveContainer" containerID="fe085febe6eba893d886b8efd1192ba33f322b6637a532d1a4d39a319fdc1360" Nov 02 19:29:35.579897 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:35.579864 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcxzs" event={"ID":"be54cb41-84fc-4d14-a8c8-0435cf0954c9","Type":"ContainerStarted","Data":"059a3da76624b53f17cfba24bd51419fedc7abf497e47ad43987a68a8f396f6f"} Nov 02 19:29:39.595267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:39.595228 4966 generic.go:356] "Generic (PLEG): container finished" podID="f16e19f1-3da4-43db-b861-7146aa9dbede" containerID="848ffa46cc1a7a438e7a5fac7276960ea3de60cfd7d910c848bed1e634d79ad0" exitCode=0 Nov 02 19:29:39.595680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:39.595322 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w8zlt" event={"ID":"f16e19f1-3da4-43db-b861-7146aa9dbede","Type":"ContainerDied","Data":"848ffa46cc1a7a438e7a5fac7276960ea3de60cfd7d910c848bed1e634d79ad0"} Nov 02 19:29:39.595680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:39.595622 4966 scope.go:122] "RemoveContainer" containerID="848ffa46cc1a7a438e7a5fac7276960ea3de60cfd7d910c848bed1e634d79ad0" Nov 02 19:29:40.599969 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:40.599935 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-w8zlt" event={"ID":"f16e19f1-3da4-43db-b861-7146aa9dbede","Type":"ContainerStarted","Data":"e971394dc9b88c6e2321336736b7f00f7956fbf8121dae77c6f1f17bf97303df"} Nov 02 19:29:44.751150 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:29:44.751117 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:29:49.632537 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:49.632454 4966 generic.go:356] "Generic (PLEG): container finished" podID="add7fff3-f01d-4f58-893f-7cf024ae59e2" containerID="8d6089fdb2e1970fb06cb44ec668dfaafa745a6aeeb50050d8c204802827d1be" exitCode=0 Nov 02 19:29:49.632537 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:49.632517 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" event={"ID":"add7fff3-f01d-4f58-893f-7cf024ae59e2","Type":"ContainerDied","Data":"8d6089fdb2e1970fb06cb44ec668dfaafa745a6aeeb50050d8c204802827d1be"} Nov 02 19:29:49.632968 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:49.632831 4966 scope.go:122] "RemoveContainer" containerID="8d6089fdb2e1970fb06cb44ec668dfaafa745a6aeeb50050d8c204802827d1be" Nov 02 19:29:50.637507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:29:50.637475 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d9l6n" event={"ID":"add7fff3-f01d-4f58-893f-7cf024ae59e2","Type":"ContainerStarted","Data":"7badf08035bb6fc978cca13ce406a03b5d0b594896a9a8f364aaa15a74ffeaab"} Nov 02 19:30:14.749870 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:30:14.749833 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:30:44.750627 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:30:44.750585 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:31:14.750815 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:31:14.750773 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:31:44.750547 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:31:44.750512 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:31:50.692229 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.692191 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv"] Nov 02 19:31:50.692718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.692574 4966 cpu_manager.go:414] "containerMap: removing container" podUID="0656367c-7665-406c-af95-d47c276692aa" containerName="registry" Nov 02 19:31:50.692718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.692593 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0656367c-7665-406c-af95-d47c276692aa" containerName="registry" Nov 02 19:31:50.692718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.692617 4966 cpu_manager.go:414] "containerMap: removing container" podUID="74b7c553-4232-4e4c-9770-494d8c427361" containerName="registry" Nov 02 19:31:50.692718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.692624 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b7c553-4232-4e4c-9770-494d8c427361" containerName="registry" Nov 02 19:31:50.692718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.692693 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0656367c-7665-406c-af95-d47c276692aa" containerName="registry" Nov 02 19:31:50.692718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.692704 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b7c553-4232-4e4c-9770-494d8c427361" containerName="registry" Nov 02 19:31:50.695820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.695801 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.728793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.728766 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.728911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.728796 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69p8\" (UniqueName: \"kubernetes.io/projected/17b3ffd7-af68-482e-b934-401ed44c55b5-kube-api-access-h69p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.728911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.728822 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.743924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.743902 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 19:31:50.744014 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.743937 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 19:31:50.791055 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.791031 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv"] Nov 02 19:31:50.802145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.802126 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8fnkt\"" Nov 02 19:31:50.829564 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.829541 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.829643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.829570 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69p8\" (UniqueName: \"kubernetes.io/projected/17b3ffd7-af68-482e-b934-401ed44c55b5-kube-api-access-h69p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.829643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.829596 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.829976 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.829960 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.830013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.829972 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:50.887854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:50.887827 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69p8\" (UniqueName: \"kubernetes.io/projected/17b3ffd7-af68-482e-b934-401ed44c55b5-kube-api-access-h69p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:51.011637 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:51.011609 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:31:51.372839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:51.372678 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv"] Nov 02 19:31:51.375419 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:51.375388 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b3ffd7_af68_482e_b934_401ed44c55b5.slice/crio-aff824c149fd26849fc7fff8b651cc23e1f5867d9d869d7f7676d5e064275b26 WatchSource:0}: Error finding container aff824c149fd26849fc7fff8b651cc23e1f5867d9d869d7f7676d5e064275b26: Status 404 returned error can't find the container with id aff824c149fd26849fc7fff8b651cc23e1f5867d9d869d7f7676d5e064275b26 Nov 02 19:31:52.052470 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:52.052433 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" event={"ID":"17b3ffd7-af68-482e-b934-401ed44c55b5","Type":"ContainerStarted","Data":"aff824c149fd26849fc7fff8b651cc23e1f5867d9d869d7f7676d5e064275b26"} Nov 02 19:31:53.194904 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.194871 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr"] Nov 02 19:31:53.199017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.198994 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.253948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.253906 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.254085 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.254053 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.254130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.254084 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99nzv\" (UniqueName: \"kubernetes.io/projected/af1f44a6-c60c-471b-a708-a616851b1059-kube-api-access-99nzv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.293659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.293622 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6jdfh\"" Nov 02 19:31:53.294673 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.294649 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr"] Nov 02 19:31:53.354808 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.354775 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.354976 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.354810 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99nzv\" (UniqueName: \"kubernetes.io/projected/af1f44a6-c60c-471b-a708-a616851b1059-kube-api-access-99nzv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.354976 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.354856 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.355273 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.355249 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.355520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.355497 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.409083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.409048 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99nzv\" (UniqueName: \"kubernetes.io/projected/af1f44a6-c60c-471b-a708-a616851b1059-kube-api-access-99nzv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.517407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.517368 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:31:53.889848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:53.889761 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr"] Nov 02 19:31:53.893761 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:53.893726 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1f44a6_c60c_471b_a708_a616851b1059.slice/crio-a2b51b93c3c49052a77b743125ff776429ab39435cdc97df45e195f618983edc WatchSource:0}: Error finding container a2b51b93c3c49052a77b743125ff776429ab39435cdc97df45e195f618983edc: Status 404 returned error can't find the container with id a2b51b93c3c49052a77b743125ff776429ab39435cdc97df45e195f618983edc Nov 02 19:31:54.061468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:54.061433 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" event={"ID":"af1f44a6-c60c-471b-a708-a616851b1059","Type":"ContainerStarted","Data":"a2b51b93c3c49052a77b743125ff776429ab39435cdc97df45e195f618983edc"} Nov 02 19:31:55.241435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.241345 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp"] Nov 02 19:31:55.246420 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.246399 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.270977 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.270944 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.271101 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.271009 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.271217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.271119 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2bf\" (UniqueName: \"kubernetes.io/projected/69ae49de-5511-4da0-b8d1-0015258bc8d9-kube-api-access-2x2bf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.341464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.340884 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-766v4\"" Nov 02 19:31:55.344847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.344816 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp"] Nov 02 19:31:55.373603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.372508 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2bf\" (UniqueName: \"kubernetes.io/projected/69ae49de-5511-4da0-b8d1-0015258bc8d9-kube-api-access-2x2bf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.373603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.372578 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.373603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.372623 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.373603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.373349 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.373603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.373566 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.388264 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.388237 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n"] Nov 02 19:31:55.392072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.392049 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.410848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.410821 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n"] Nov 02 19:31:55.421024 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.420995 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5"] Nov 02 19:31:55.425623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.425599 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.428564 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.428534 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2bf\" (UniqueName: \"kubernetes.io/projected/69ae49de-5511-4da0-b8d1-0015258bc8d9-kube-api-access-2x2bf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.440582 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.440555 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-h6pmr\"" Nov 02 19:31:55.473696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.473622 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.473868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.473844 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfktr\" (UniqueName: \"kubernetes.io/projected/7798e0a7-c083-4e6e-8e24-194000cfa12b-kube-api-access-bfktr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.473984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.473947 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.474115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.473999 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.474115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.474032 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.474228 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.474147 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6kbq\" (UniqueName: \"kubernetes.io/projected/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-kube-api-access-q6kbq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.477090 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.477063 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-q9s4d\"" Nov 02 19:31:55.479101 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.479073 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5"] Nov 02 19:31:55.521436 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.521355 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89"] Nov 02 19:31:55.524948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.524925 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.565424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.565398 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:31:55.573285 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.573258 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-vs66w\"" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574640 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574677 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574708 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574741 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574771 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574805 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/68d7c87b-1ede-4b8e-8b66-523d832063f8-kube-api-access-sjlcr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574849 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6kbq\" (UniqueName: \"kubernetes.io/projected/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-kube-api-access-q6kbq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574910 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574968 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfktr\" (UniqueName: \"kubernetes.io/projected/7798e0a7-c083-4e6e-8e24-194000cfa12b-kube-api-access-bfktr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.574975 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.575230 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.575399 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.575711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.575480 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.579668 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.579640 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89"] Nov 02 19:31:55.628761 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.628729 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6kbq\" (UniqueName: \"kubernetes.io/projected/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-kube-api-access-q6kbq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.635709 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.635054 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfktr\" (UniqueName: \"kubernetes.io/projected/7798e0a7-c083-4e6e-8e24-194000cfa12b-kube-api-access-bfktr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.675993 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.675964 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.676156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.676016 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.676156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.676043 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/68d7c87b-1ede-4b8e-8b66-523d832063f8-kube-api-access-sjlcr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.676818 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.676795 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.677059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.677036 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.711689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.711662 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:31:55.730253 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.730223 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/68d7c87b-1ede-4b8e-8b66-523d832063f8-kube-api-access-sjlcr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.754901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.754844 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:31:55.843869 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.843623 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:31:55.950411 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:55.950370 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp"] Nov 02 19:31:55.955586 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:55.955550 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ae49de_5511_4da0_b8d1_0015258bc8d9.slice/crio-3c51d2e9290813e373cb9c4051ea61bec9237c4d3ed6e2bbdfa3a14582306a51 WatchSource:0}: Error finding container 3c51d2e9290813e373cb9c4051ea61bec9237c4d3ed6e2bbdfa3a14582306a51: Status 404 returned error can't find the container with id 3c51d2e9290813e373cb9c4051ea61bec9237c4d3ed6e2bbdfa3a14582306a51 Nov 02 19:31:56.074712 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.074595 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" event={"ID":"69ae49de-5511-4da0-b8d1-0015258bc8d9","Type":"ContainerStarted","Data":"3c51d2e9290813e373cb9c4051ea61bec9237c4d3ed6e2bbdfa3a14582306a51"} Nov 02 19:31:56.085286 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.085258 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n"] Nov 02 19:31:56.089603 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:56.089573 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3c9855_9a5a_4c7e_9f7f_343e276f7e74.slice/crio-fb6d568d509afc46f5c806838cf074226fd4c9cd419e84cd224b0dbaedef0d56 WatchSource:0}: Error finding container fb6d568d509afc46f5c806838cf074226fd4c9cd419e84cd224b0dbaedef0d56: Status 404 returned error can't find the container with id fb6d568d509afc46f5c806838cf074226fd4c9cd419e84cd224b0dbaedef0d56 Nov 02 19:31:56.134424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.134396 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5"] Nov 02 19:31:56.137136 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:56.137093 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7798e0a7_c083_4e6e_8e24_194000cfa12b.slice/crio-8e647bfafd52e20d74c6c9a49353ffa2f76063d1cb9c4d83db102a4040283807 WatchSource:0}: Error finding container 8e647bfafd52e20d74c6c9a49353ffa2f76063d1cb9c4d83db102a4040283807: Status 404 returned error can't find the container with id 8e647bfafd52e20d74c6c9a49353ffa2f76063d1cb9c4d83db102a4040283807 Nov 02 19:31:56.223326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.223266 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89"] Nov 02 19:31:56.227424 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:56.227396 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d7c87b_1ede_4b8e_8b66_523d832063f8.slice/crio-28261af67cb57942c7dafb592fef9421c33cda4b8b0d606aa0c7ad5c2db7590a WatchSource:0}: Error finding container 28261af67cb57942c7dafb592fef9421c33cda4b8b0d606aa0c7ad5c2db7590a: Status 404 returned error can't find the container with id 28261af67cb57942c7dafb592fef9421c33cda4b8b0d606aa0c7ad5c2db7590a Nov 02 19:31:56.892320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.892227 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn"] Nov 02 19:31:56.896749 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.896725 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:56.945632 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.945588 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-79lwf\"" Nov 02 19:31:56.990592 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.990558 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:56.990752 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.990605 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lssb\" (UniqueName: \"kubernetes.io/projected/95677672-f5dd-49d1-8746-e4aa52eb6d8c-kube-api-access-7lssb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:56.990752 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.990673 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:56.993071 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:56.993034 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn"] Nov 02 19:31:57.086373 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.086333 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" event={"ID":"68d7c87b-1ede-4b8e-8b66-523d832063f8","Type":"ContainerStarted","Data":"28261af67cb57942c7dafb592fef9421c33cda4b8b0d606aa0c7ad5c2db7590a"} Nov 02 19:31:57.088062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.088011 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" event={"ID":"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74","Type":"ContainerStarted","Data":"fb6d568d509afc46f5c806838cf074226fd4c9cd419e84cd224b0dbaedef0d56"} Nov 02 19:31:57.090441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.090410 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" event={"ID":"7798e0a7-c083-4e6e-8e24-194000cfa12b","Type":"ContainerStarted","Data":"8e647bfafd52e20d74c6c9a49353ffa2f76063d1cb9c4d83db102a4040283807"} Nov 02 19:31:57.091245 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.091220 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:57.091361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.091263 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lssb\" (UniqueName: \"kubernetes.io/projected/95677672-f5dd-49d1-8746-e4aa52eb6d8c-kube-api-access-7lssb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:57.091361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.091328 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:57.091790 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.091769 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:57.092051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.092031 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:57.161279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.161212 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lssb\" (UniqueName: \"kubernetes.io/projected/95677672-f5dd-49d1-8746-e4aa52eb6d8c-kube-api-access-7lssb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:57.218435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.218403 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:31:57.625824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:57.625782 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn"] Nov 02 19:31:58.100010 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.099933 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" event={"ID":"95677672-f5dd-49d1-8746-e4aa52eb6d8c","Type":"ContainerStarted","Data":"6817aab1a39c17bde4882eff2607973c19f49aa3829497bb83aa7e749991a706"} Nov 02 19:31:58.360400 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.358620 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn"] Nov 02 19:31:58.364273 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.364244 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.406154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.406120 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfn2g\" (UniqueName: \"kubernetes.io/projected/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-kube-api-access-lfn2g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.406351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.406193 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.406426 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.406391 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.420760 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.419190 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn"] Nov 02 19:31:58.507456 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.507416 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.507647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.507515 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfn2g\" (UniqueName: \"kubernetes.io/projected/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-kube-api-access-lfn2g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.507647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.507572 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.508059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.508038 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.508352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.508330 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.572336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.572153 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfn2g\" (UniqueName: \"kubernetes.io/projected/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-kube-api-access-lfn2g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.688623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.688289 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:31:58.833751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.829344 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb"] Nov 02 19:31:58.834417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.834392 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.886609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.885934 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 19:31:58.907237 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.907033 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qmwl4\"" Nov 02 19:31:58.912374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912221 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912279 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912423 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912459 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912562 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912605 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6p4\" (UniqueName: \"kubernetes.io/projected/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-kube-api-access-gm6p4\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912644 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912689 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.912755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.912715 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:58.933927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:58.933791 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb"] Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.013716 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.013765 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.013801 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.013839 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6p4\" (UniqueName: \"kubernetes.io/projected/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-kube-api-access-gm6p4\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.013947 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.014018 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.014045 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.014126 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.014160 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.014754 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.014996 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.015943 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.016783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.016730 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.037582 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.016995 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.037582 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.018220 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.037582 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.020020 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.070386 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.069680 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.073402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.073267 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6p4\" (UniqueName: \"kubernetes.io/projected/c135774f-3f52-4d4c-81cb-aa2c86ea37d5-kube-api-access-gm6p4\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-54rpb\" (UID: \"c135774f-3f52-4d4c-81cb-aa2c86ea37d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.124485 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.122812 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm"] Nov 02 19:31:59.128195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.128170 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.139797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.139639 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn"] Nov 02 19:31:59.147859 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:59.147820 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c4ec3c_5fe5_47b4_8630_2520bd7bab6a.slice/crio-89d823538b994d3d46a6e224f2410ed7374ee584dd10040bd4551390ae309987 WatchSource:0}: Error finding container 89d823538b994d3d46a6e224f2410ed7374ee584dd10040bd4551390ae309987: Status 404 returned error can't find the container with id 89d823538b994d3d46a6e224f2410ed7374ee584dd10040bd4551390ae309987 Nov 02 19:31:59.164379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.162377 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:31:59.177490 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.177019 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-httgp\"" Nov 02 19:31:59.185727 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.185242 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm"] Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.215762 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.215830 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.215867 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvl8k\" (UniqueName: \"kubernetes.io/projected/8c852f75-6a1d-4521-a67f-20fd5db57355-kube-api-access-cvl8k\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.215923 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.215955 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.216012 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.216036 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.216073 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c852f75-6a1d-4521-a67f-20fd5db57355-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.216363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.216115 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.316897 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.316832 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c852f75-6a1d-4521-a67f-20fd5db57355-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.316916 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.316972 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.317009 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.317035 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvl8k\" (UniqueName: \"kubernetes.io/projected/8c852f75-6a1d-4521-a67f-20fd5db57355-kube-api-access-cvl8k\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.317089 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.317121 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.317188 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.317218 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.317668 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.317650 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.318344 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.318316 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c852f75-6a1d-4521-a67f-20fd5db57355-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.318608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.318586 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.320243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.319329 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.320243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.320176 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.323038 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.322907 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.324664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.324640 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.373641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.373590 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvl8k\" (UniqueName: \"kubernetes.io/projected/8c852f75-6a1d-4521-a67f-20fd5db57355-kube-api-access-cvl8k\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.389327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.389270 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c852f75-6a1d-4521-a67f-20fd5db57355-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm\" (UID: \"8c852f75-6a1d-4521-a67f-20fd5db57355\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.466072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.465770 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:31:59.606045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.605947 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb"] Nov 02 19:31:59.890322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:31:59.889054 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm"] Nov 02 19:31:59.901400 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:31:59.901368 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c852f75_6a1d_4521_a67f_20fd5db57355.slice/crio-c95296049eafd1e7c8710912a44adf7fe3c5444e3fadd022f75d9a0341078623 WatchSource:0}: Error finding container c95296049eafd1e7c8710912a44adf7fe3c5444e3fadd022f75d9a0341078623: Status 404 returned error can't find the container with id c95296049eafd1e7c8710912a44adf7fe3c5444e3fadd022f75d9a0341078623 Nov 02 19:32:00.137151 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:00.136374 4966 generic.go:356] "Generic (PLEG): container finished" podID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerID="7390da9f11921679126a71f64d7628f5f21c169e312855f8e472daff9dbdf3d2" exitCode=0 Nov 02 19:32:00.137151 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:00.136476 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" event={"ID":"17b3ffd7-af68-482e-b934-401ed44c55b5","Type":"ContainerDied","Data":"7390da9f11921679126a71f64d7628f5f21c169e312855f8e472daff9dbdf3d2"} Nov 02 19:32:00.144489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:00.143732 4966 generic.go:356] "Generic (PLEG): container finished" podID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerID="c6b747a235785ede835f8c98f8b0954ce0bc96fc6e406f9339ae24157f536a33" exitCode=0 Nov 02 19:32:00.144489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:00.143831 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" event={"ID":"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a","Type":"ContainerDied","Data":"c6b747a235785ede835f8c98f8b0954ce0bc96fc6e406f9339ae24157f536a33"} Nov 02 19:32:00.144489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:00.143859 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" event={"ID":"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a","Type":"ContainerStarted","Data":"89d823538b994d3d46a6e224f2410ed7374ee584dd10040bd4551390ae309987"} Nov 02 19:32:00.150721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:00.150696 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" event={"ID":"8c852f75-6a1d-4521-a67f-20fd5db57355","Type":"ContainerStarted","Data":"c95296049eafd1e7c8710912a44adf7fe3c5444e3fadd022f75d9a0341078623"} Nov 02 19:32:00.152848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:00.152828 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" event={"ID":"c135774f-3f52-4d4c-81cb-aa2c86ea37d5","Type":"ContainerStarted","Data":"026346bfd3ba029d5719b5b2110a5e68ddf275a81d8da515c59c40bd22e3e056"} Nov 02 19:32:01.162895 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:01.162861 4966 generic.go:356] "Generic (PLEG): container finished" podID="af1f44a6-c60c-471b-a708-a616851b1059" containerID="5483d35dabc338cd6a0ccd65ee811ca2f6978c97e9098a6e7d9d2d462d8e9d47" exitCode=0 Nov 02 19:32:01.163402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:01.162979 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" event={"ID":"af1f44a6-c60c-471b-a708-a616851b1059","Type":"ContainerDied","Data":"5483d35dabc338cd6a0ccd65ee811ca2f6978c97e9098a6e7d9d2d462d8e9d47"} Nov 02 19:32:03.209641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:03.209561 4966 generic.go:356] "Generic (PLEG): container finished" podID="af1f44a6-c60c-471b-a708-a616851b1059" containerID="493c3ad57a3c1810794b42e752a103893e223f6037f73bf95881778a28f80c1e" exitCode=0 Nov 02 19:32:03.210100 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:03.209661 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" event={"ID":"af1f44a6-c60c-471b-a708-a616851b1059","Type":"ContainerDied","Data":"493c3ad57a3c1810794b42e752a103893e223f6037f73bf95881778a28f80c1e"} Nov 02 19:32:05.274454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:05.274353 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" event={"ID":"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74","Type":"ContainerStarted","Data":"8f97c893579959d6b503fa3fcdb39d0cccf14fca618c2d475d4cd62eb1ad7f61"} Nov 02 19:32:06.284041 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.283981 4966 generic.go:356] "Generic (PLEG): container finished" podID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerID="8f97c893579959d6b503fa3fcdb39d0cccf14fca618c2d475d4cd62eb1ad7f61" exitCode=0 Nov 02 19:32:06.284552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.284077 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" event={"ID":"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74","Type":"ContainerDied","Data":"8f97c893579959d6b503fa3fcdb39d0cccf14fca618c2d475d4cd62eb1ad7f61"} Nov 02 19:32:06.287146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.287119 4966 generic.go:356] "Generic (PLEG): container finished" podID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerID="e6eee8aba19173246677064109d232192cf6f738b846711c690dc5993062f870" exitCode=0 Nov 02 19:32:06.287265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.287244 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" event={"ID":"7798e0a7-c083-4e6e-8e24-194000cfa12b","Type":"ContainerDied","Data":"e6eee8aba19173246677064109d232192cf6f738b846711c690dc5993062f870"} Nov 02 19:32:06.289079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.289053 4966 generic.go:356] "Generic (PLEG): container finished" podID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerID="f3b8d21d22d93a594f31712902bad92b72c606a00e7ade74e8317baef1d54150" exitCode=0 Nov 02 19:32:06.289176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.289146 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" event={"ID":"68d7c87b-1ede-4b8e-8b66-523d832063f8","Type":"ContainerDied","Data":"f3b8d21d22d93a594f31712902bad92b72c606a00e7ade74e8317baef1d54150"} Nov 02 19:32:06.292037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.292010 4966 generic.go:356] "Generic (PLEG): container finished" podID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerID="f0eb02726b7f392ba6ad1aa50ef4a44fb0a66490d6497e84da69ee3bab02e1ba" exitCode=0 Nov 02 19:32:06.292140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:06.292051 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" event={"ID":"69ae49de-5511-4da0-b8d1-0015258bc8d9","Type":"ContainerDied","Data":"f0eb02726b7f392ba6ad1aa50ef4a44fb0a66490d6497e84da69ee3bab02e1ba"} Nov 02 19:32:07.302602 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.302565 4966 generic.go:356] "Generic (PLEG): container finished" podID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerID="d7097807b68966d23856440977cf3be8c4299509dea61ab913f51f9905ee658b" exitCode=0 Nov 02 19:32:07.303075 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.302820 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" event={"ID":"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a","Type":"ContainerDied","Data":"d7097807b68966d23856440977cf3be8c4299509dea61ab913f51f9905ee658b"} Nov 02 19:32:07.306439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.305967 4966 generic.go:356] "Generic (PLEG): container finished" podID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerID="b72313aa6a7dd0928af29f71f49a5517ca6fbeb61aa8de3fa3147821f9fc8e86" exitCode=0 Nov 02 19:32:07.306439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.306159 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" event={"ID":"95677672-f5dd-49d1-8746-e4aa52eb6d8c","Type":"ContainerDied","Data":"b72313aa6a7dd0928af29f71f49a5517ca6fbeb61aa8de3fa3147821f9fc8e86"} Nov 02 19:32:07.309901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.309877 4966 generic.go:356] "Generic (PLEG): container finished" podID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerID="a5497ac4abac761e687e789a8b539d6627357c15f1a690f58740ce955a88c911" exitCode=0 Nov 02 19:32:07.309997 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.309919 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" event={"ID":"17b3ffd7-af68-482e-b934-401ed44c55b5","Type":"ContainerDied","Data":"a5497ac4abac761e687e789a8b539d6627357c15f1a690f58740ce955a88c911"} Nov 02 19:32:07.590988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.590958 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:32:07.591114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.591030 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:32:07.591114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.591068 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:32:07.843016 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.842972 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:32:07.843129 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.843071 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:32:07.843178 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:07.843123 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:32:08.319765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:08.319727 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" event={"ID":"8c852f75-6a1d-4521-a67f-20fd5db57355","Type":"ContainerStarted","Data":"26dfffbbc67c3bd85deb7ae99e479022003acba4d10cabf582de397e1e49e94d"} Nov 02 19:32:08.323441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:08.323416 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" event={"ID":"7798e0a7-c083-4e6e-8e24-194000cfa12b","Type":"ContainerStarted","Data":"c746abb1930331db1234eb83315a5179d19b3ea7c9248e4e5fff98c225ab8958"} Nov 02 19:32:08.329683 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:08.328194 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" event={"ID":"c135774f-3f52-4d4c-81cb-aa2c86ea37d5","Type":"ContainerStarted","Data":"6ad127fcf2866f74406f343d7db81e8b32778e84ffa49e0c9d35021b0e8249d7"} Nov 02 19:32:08.430024 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:08.429970 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" podStartSLOduration=1.492824632 podStartE2EDuration="9.429952905s" totalImagesPullingTime="7.937128273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:08.428666157 +0000 UTC m=+234.377321998" watchObservedRunningTime="2026-11-02 19:32:08.429952905 +0000 UTC m=+234.378608746" Nov 02 19:32:08.468326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:08.467215 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:32:08.472894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:08.472871 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:32:08.543492 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:08.543431 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" podStartSLOduration=2.579426323 podStartE2EDuration="10.543415098s" totalImagesPullingTime="7.963988775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:08.542800897 +0000 UTC m=+234.491456736" watchObservedRunningTime="2026-11-02 19:32:08.543415098 +0000 UTC m=+234.492070938" Nov 02 19:32:09.166770 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.166736 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:32:09.173434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.173157 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:32:09.339471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.338476 4966 generic.go:356] "Generic (PLEG): container finished" podID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerID="a8ba14179bc55d5a4d43e1ea5c79065ce336505ab021a527af21a8641bb03705" exitCode=0 Nov 02 19:32:09.339471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.338588 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" event={"ID":"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74","Type":"ContainerDied","Data":"a8ba14179bc55d5a4d43e1ea5c79065ce336505ab021a527af21a8641bb03705"} Nov 02 19:32:09.345874 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.345702 4966 generic.go:356] "Generic (PLEG): container finished" podID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerID="c746abb1930331db1234eb83315a5179d19b3ea7c9248e4e5fff98c225ab8958" exitCode=0 Nov 02 19:32:09.345874 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.345813 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" event={"ID":"7798e0a7-c083-4e6e-8e24-194000cfa12b","Type":"ContainerDied","Data":"c746abb1930331db1234eb83315a5179d19b3ea7c9248e4e5fff98c225ab8958"} Nov 02 19:32:09.350958 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.350276 4966 generic.go:356] "Generic (PLEG): container finished" podID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerID="0bd2950f296ad7e1f1d56cb43aa63d0e0fbd821e17e55f85d03f05115d81e665" exitCode=0 Nov 02 19:32:09.350958 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.350379 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" event={"ID":"68d7c87b-1ede-4b8e-8b66-523d832063f8","Type":"ContainerDied","Data":"0bd2950f296ad7e1f1d56cb43aa63d0e0fbd821e17e55f85d03f05115d81e665"} Nov 02 19:32:09.355576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.353547 4966 generic.go:356] "Generic (PLEG): container finished" podID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerID="d6f6daa628da4782dbce87bf2a71f3c55685e28be067213c677f6340360ea834" exitCode=0 Nov 02 19:32:09.355576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.354741 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" event={"ID":"69ae49de-5511-4da0-b8d1-0015258bc8d9","Type":"ContainerDied","Data":"d6f6daa628da4782dbce87bf2a71f3c55685e28be067213c677f6340360ea834"} Nov 02 19:32:09.355576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.354774 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:32:09.355576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.355518 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:32:09.356387 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.356244 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm" Nov 02 19:32:09.356387 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:09.356363 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:32:12.387967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:12.387930 4966 generic.go:356] "Generic (PLEG): container finished" podID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerID="a71de851e967cb0b204205d0926b0e3f25f12935823679f84f24380cc2ae5978" exitCode=0 Nov 02 19:32:12.388547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:12.388065 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" event={"ID":"95677672-f5dd-49d1-8746-e4aa52eb6d8c","Type":"ContainerDied","Data":"a71de851e967cb0b204205d0926b0e3f25f12935823679f84f24380cc2ae5978"} Nov 02 19:32:14.785291 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:32:14.785250 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:32:19.480086 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:19.480022 4966 generic.go:356] "Generic (PLEG): container finished" podID="af1f44a6-c60c-471b-a708-a616851b1059" containerID="4107c6d08bd59db5aafea80acf3afa1b3b298a863f76a997f6dc76ea3a35baec" exitCode=0 Nov 02 19:32:19.480086 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:19.480087 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" event={"ID":"af1f44a6-c60c-471b-a708-a616851b1059","Type":"ContainerDied","Data":"4107c6d08bd59db5aafea80acf3afa1b3b298a863f76a997f6dc76ea3a35baec"} Nov 02 19:32:20.490004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.489918 4966 generic.go:356] "Generic (PLEG): container finished" podID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerID="b1cb382450583037f549d13cb36e26a1f562b9a53c8e70fed2a759f02183e9fd" exitCode=0 Nov 02 19:32:20.490492 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.490020 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" event={"ID":"68d7c87b-1ede-4b8e-8b66-523d832063f8","Type":"ContainerDied","Data":"b1cb382450583037f549d13cb36e26a1f562b9a53c8e70fed2a759f02183e9fd"} Nov 02 19:32:20.493638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.493604 4966 generic.go:356] "Generic (PLEG): container finished" podID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerID="2d839346bd9beeeb565bd8b53f197b394a59e1cdd6a2be15be25902e9ad75151" exitCode=0 Nov 02 19:32:20.493777 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.493736 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" event={"ID":"69ae49de-5511-4da0-b8d1-0015258bc8d9","Type":"ContainerDied","Data":"2d839346bd9beeeb565bd8b53f197b394a59e1cdd6a2be15be25902e9ad75151"} Nov 02 19:32:20.497388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.497357 4966 generic.go:356] "Generic (PLEG): container finished" podID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerID="7a0158b71146aa4cb658c997a71aa2be5c58212a391eb11889629abae78b8b37" exitCode=0 Nov 02 19:32:20.497592 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.497568 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" event={"ID":"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74","Type":"ContainerDied","Data":"7a0158b71146aa4cb658c997a71aa2be5c58212a391eb11889629abae78b8b37"} Nov 02 19:32:20.500918 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.500857 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" event={"ID":"7798e0a7-c083-4e6e-8e24-194000cfa12b","Type":"ContainerStarted","Data":"2186165a8f9c2f250d37bcdc0108c075171d563118ac38af52fafac77fca4365"} Nov 02 19:32:20.663216 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.663162 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:32:20.750783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.750689 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-util\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-util\") pod \"af1f44a6-c60c-471b-a708-a616851b1059\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " Nov 02 19:32:20.750783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.750763 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-bundle\") pod \"af1f44a6-c60c-471b-a708-a616851b1059\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " Nov 02 19:32:20.751027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.750831 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1f44a6-c60c-471b-a708-a616851b1059-kube-api-access-99nzv\" (UniqueName: \"kubernetes.io/projected/af1f44a6-c60c-471b-a708-a616851b1059-kube-api-access-99nzv\") pod \"af1f44a6-c60c-471b-a708-a616851b1059\" (UID: \"af1f44a6-c60c-471b-a708-a616851b1059\") " Nov 02 19:32:20.751460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.751426 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-bundle" pod "af1f44a6-c60c-471b-a708-a616851b1059" (UID: "af1f44a6-c60c-471b-a708-a616851b1059"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:20.754487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.754457 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1f44a6-c60c-471b-a708-a616851b1059-kube-api-access-99nzv" pod "af1f44a6-c60c-471b-a708-a616851b1059" (UID: "af1f44a6-c60c-471b-a708-a616851b1059"). InnerVolumeSpecName "kube-api-access-99nzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:20.757080 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.757051 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-util" pod "af1f44a6-c60c-471b-a708-a616851b1059" (UID: "af1f44a6-c60c-471b-a708-a616851b1059"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:20.852015 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.851980 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:20.852015 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.852020 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af1f44a6-c60c-471b-a708-a616851b1059-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:20.852259 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.852034 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99nzv\" (UniqueName: \"kubernetes.io/projected/af1f44a6-c60c-471b-a708-a616851b1059-kube-api-access-99nzv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:20.928799 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:20.928734 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" podStartSLOduration=3.775903386 podStartE2EDuration="25.928713521s" totalImagesPullingTime="22.152810135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:31:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:20.926528917 +0000 UTC m=+246.875184748" watchObservedRunningTime="2026-11-02 19:32:20.928713521 +0000 UTC m=+246.877369360" Nov 02 19:32:21.512360 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.507936 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" event={"ID":"af1f44a6-c60c-471b-a708-a616851b1059","Type":"ContainerDied","Data":"a2b51b93c3c49052a77b743125ff776429ab39435cdc97df45e195f618983edc"} Nov 02 19:32:21.512360 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.508000 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2b51b93c3c49052a77b743125ff776429ab39435cdc97df45e195f618983edc" Nov 02 19:32:21.512360 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.508073 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nqjbr" Nov 02 19:32:21.519366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.519327 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" event={"ID":"95677672-f5dd-49d1-8746-e4aa52eb6d8c","Type":"ContainerStarted","Data":"c3e744306d47d9253a1dbba9329d3a5d2bf952b5f491f37eb39a3ff80b83e429"} Nov 02 19:32:21.522476 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.522450 4966 generic.go:356] "Generic (PLEG): container finished" podID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerID="2186165a8f9c2f250d37bcdc0108c075171d563118ac38af52fafac77fca4365" exitCode=0 Nov 02 19:32:21.522608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.522555 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" event={"ID":"7798e0a7-c083-4e6e-8e24-194000cfa12b","Type":"ContainerDied","Data":"2186165a8f9c2f250d37bcdc0108c075171d563118ac38af52fafac77fca4365"} Nov 02 19:32:21.629134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.627600 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" podStartSLOduration=2.915163219 podStartE2EDuration="25.627582685s" totalImagesPullingTime="22.712419466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:31:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:21.625340398 +0000 UTC m=+247.573996237" watchObservedRunningTime="2026-11-02 19:32:21.627582685 +0000 UTC m=+247.576238522" Nov 02 19:32:21.688437 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.683924 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:32:21.731085 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.731058 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:32:21.750195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.750172 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:32:21.760713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.760691 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-kube-api-access-q6kbq\" (UniqueName: \"kubernetes.io/projected/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-kube-api-access-q6kbq\") pod \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " Nov 02 19:32:21.760825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.760811 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-util\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-util\") pod \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " Nov 02 19:32:21.760893 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.760863 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-bundle\") pod \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\" (UID: \"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74\") " Nov 02 19:32:21.761602 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.761572 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-bundle" pod "ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" (UID: "ee3c9855-9a5a-4c7e-9f7f-343e276f7e74"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:21.762747 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.762699 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-kube-api-access-q6kbq" pod "ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" (UID: "ee3c9855-9a5a-4c7e-9f7f-343e276f7e74"). InnerVolumeSpecName "kube-api-access-q6kbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:21.765061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.765042 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-util" pod "ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" (UID: "ee3c9855-9a5a-4c7e-9f7f-343e276f7e74"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:21.862327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862279 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d7c87b-1ede-4b8e-8b66-523d832063f8-kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/68d7c87b-1ede-4b8e-8b66-523d832063f8-kube-api-access-sjlcr\") pod \"68d7c87b-1ede-4b8e-8b66-523d832063f8\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " Nov 02 19:32:21.862487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862355 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-util\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-util\") pod \"68d7c87b-1ede-4b8e-8b66-523d832063f8\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " Nov 02 19:32:21.862487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862426 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-bundle\") pod \"68d7c87b-1ede-4b8e-8b66-523d832063f8\" (UID: \"68d7c87b-1ede-4b8e-8b66-523d832063f8\") " Nov 02 19:32:21.862593 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862484 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-util\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-util\") pod \"69ae49de-5511-4da0-b8d1-0015258bc8d9\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " Nov 02 19:32:21.862593 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862526 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ae49de-5511-4da0-b8d1-0015258bc8d9-kube-api-access-2x2bf\" (UniqueName: \"kubernetes.io/projected/69ae49de-5511-4da0-b8d1-0015258bc8d9-kube-api-access-2x2bf\") pod \"69ae49de-5511-4da0-b8d1-0015258bc8d9\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " Nov 02 19:32:21.862706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862622 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-bundle\") pod \"69ae49de-5511-4da0-b8d1-0015258bc8d9\" (UID: \"69ae49de-5511-4da0-b8d1-0015258bc8d9\") " Nov 02 19:32:21.862833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862814 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6kbq\" (UniqueName: \"kubernetes.io/projected/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-kube-api-access-q6kbq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.862901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862837 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.862901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.862852 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee3c9855-9a5a-4c7e-9f7f-343e276f7e74-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.863132 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.863082 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-bundle" pod "68d7c87b-1ede-4b8e-8b66-523d832063f8" (UID: "68d7c87b-1ede-4b8e-8b66-523d832063f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:21.863387 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.863360 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-bundle" pod "69ae49de-5511-4da0-b8d1-0015258bc8d9" (UID: "69ae49de-5511-4da0-b8d1-0015258bc8d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:21.865001 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.864975 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d7c87b-1ede-4b8e-8b66-523d832063f8-kube-api-access-sjlcr" pod "68d7c87b-1ede-4b8e-8b66-523d832063f8" (UID: "68d7c87b-1ede-4b8e-8b66-523d832063f8"). InnerVolumeSpecName "kube-api-access-sjlcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:21.865145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.865127 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ae49de-5511-4da0-b8d1-0015258bc8d9-kube-api-access-2x2bf" pod "69ae49de-5511-4da0-b8d1-0015258bc8d9" (UID: "69ae49de-5511-4da0-b8d1-0015258bc8d9"). InnerVolumeSpecName "kube-api-access-2x2bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:21.870133 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.870102 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-util" pod "69ae49de-5511-4da0-b8d1-0015258bc8d9" (UID: "69ae49de-5511-4da0-b8d1-0015258bc8d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:21.870779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.870750 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-util" pod "68d7c87b-1ede-4b8e-8b66-523d832063f8" (UID: "68d7c87b-1ede-4b8e-8b66-523d832063f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:21.963451 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.963423 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.963451 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.963450 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2bf\" (UniqueName: \"kubernetes.io/projected/69ae49de-5511-4da0-b8d1-0015258bc8d9-kube-api-access-2x2bf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.963626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.963460 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69ae49de-5511-4da0-b8d1-0015258bc8d9-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.963626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.963469 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjlcr\" (UniqueName: \"kubernetes.io/projected/68d7c87b-1ede-4b8e-8b66-523d832063f8-kube-api-access-sjlcr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.963626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.963477 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:21.963626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:21.963487 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68d7c87b-1ede-4b8e-8b66-523d832063f8-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:22.529109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.529083 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" Nov 02 19:32:22.529109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.529101 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab8t2n" event={"ID":"ee3c9855-9a5a-4c7e-9f7f-343e276f7e74","Type":"ContainerDied","Data":"fb6d568d509afc46f5c806838cf074226fd4c9cd419e84cd224b0dbaedef0d56"} Nov 02 19:32:22.529607 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.529152 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb6d568d509afc46f5c806838cf074226fd4c9cd419e84cd224b0dbaedef0d56" Nov 02 19:32:22.531587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.531561 4966 generic.go:356] "Generic (PLEG): container finished" podID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerID="22a93ff0f47b87885068d60e754750a3bfac4a7a16e786cd75734423147b6df6" exitCode=0 Nov 02 19:32:22.531691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.531651 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" event={"ID":"17b3ffd7-af68-482e-b934-401ed44c55b5","Type":"ContainerDied","Data":"22a93ff0f47b87885068d60e754750a3bfac4a7a16e786cd75734423147b6df6"} Nov 02 19:32:22.533495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.533478 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" Nov 02 19:32:22.533599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.533477 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gmr89" event={"ID":"68d7c87b-1ede-4b8e-8b66-523d832063f8","Type":"ContainerDied","Data":"28261af67cb57942c7dafb592fef9421c33cda4b8b0d606aa0c7ad5c2db7590a"} Nov 02 19:32:22.533599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.533581 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28261af67cb57942c7dafb592fef9421c33cda4b8b0d606aa0c7ad5c2db7590a" Nov 02 19:32:22.535052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.535032 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:32:22.535886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.535866 4966 generic.go:356] "Generic (PLEG): container finished" podID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerID="3dc14180b6afa9f8e3793204e9907b8679354a39e0ab9dbb0e0a187a3802a5be" exitCode=1 Nov 02 19:32:22.535984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.535946 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" event={"ID":"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a","Type":"ContainerDied","Data":"3dc14180b6afa9f8e3793204e9907b8679354a39e0ab9dbb0e0a187a3802a5be"} Nov 02 19:32:22.537746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.537724 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" event={"ID":"69ae49de-5511-4da0-b8d1-0015258bc8d9","Type":"ContainerDied","Data":"3c51d2e9290813e373cb9c4051ea61bec9237c4d3ed6e2bbdfa3a14582306a51"} Nov 02 19:32:22.537834 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.537761 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c51d2e9290813e373cb9c4051ea61bec9237c4d3ed6e2bbdfa3a14582306a51" Nov 02 19:32:22.537834 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.537728 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp85wp" Nov 02 19:32:22.540004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.539980 4966 generic.go:356] "Generic (PLEG): container finished" podID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerID="c3e744306d47d9253a1dbba9329d3a5d2bf952b5f491f37eb39a3ff80b83e429" exitCode=0 Nov 02 19:32:22.540117 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.540057 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" event={"ID":"95677672-f5dd-49d1-8746-e4aa52eb6d8c","Type":"ContainerDied","Data":"c3e744306d47d9253a1dbba9329d3a5d2bf952b5f491f37eb39a3ff80b83e429"} Nov 02 19:32:22.662659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.662633 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:32:22.770512 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.770483 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7798e0a7-c083-4e6e-8e24-194000cfa12b-kube-api-access-bfktr\" (UniqueName: \"kubernetes.io/projected/7798e0a7-c083-4e6e-8e24-194000cfa12b-kube-api-access-bfktr\") pod \"7798e0a7-c083-4e6e-8e24-194000cfa12b\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " Nov 02 19:32:22.770651 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.770576 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-bundle\") pod \"7798e0a7-c083-4e6e-8e24-194000cfa12b\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " Nov 02 19:32:22.770651 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.770627 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-util\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-util\") pod \"7798e0a7-c083-4e6e-8e24-194000cfa12b\" (UID: \"7798e0a7-c083-4e6e-8e24-194000cfa12b\") " Nov 02 19:32:22.771091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.771064 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-bundle" pod "7798e0a7-c083-4e6e-8e24-194000cfa12b" (UID: "7798e0a7-c083-4e6e-8e24-194000cfa12b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:22.772677 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.772654 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7798e0a7-c083-4e6e-8e24-194000cfa12b-kube-api-access-bfktr" pod "7798e0a7-c083-4e6e-8e24-194000cfa12b" (UID: "7798e0a7-c083-4e6e-8e24-194000cfa12b"). InnerVolumeSpecName "kube-api-access-bfktr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:22.775536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.775512 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-util" pod "7798e0a7-c083-4e6e-8e24-194000cfa12b" (UID: "7798e0a7-c083-4e6e-8e24-194000cfa12b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:22.871561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.871532 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:22.871561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.871555 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7798e0a7-c083-4e6e-8e24-194000cfa12b-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:22.871561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:22.871565 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfktr\" (UniqueName: \"kubernetes.io/projected/7798e0a7-c083-4e6e-8e24-194000cfa12b-kube-api-access-bfktr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.546854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.546817 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" Nov 02 19:32:23.547292 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.546863 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m4lz5" event={"ID":"7798e0a7-c083-4e6e-8e24-194000cfa12b","Type":"ContainerDied","Data":"8e647bfafd52e20d74c6c9a49353ffa2f76063d1cb9c4d83db102a4040283807"} Nov 02 19:32:23.547292 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.546911 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e647bfafd52e20d74c6c9a49353ffa2f76063d1cb9c4d83db102a4040283807" Nov 02 19:32:23.697664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.697638 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:32:23.727889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.727862 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:32:23.730972 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.730952 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:32:23.733554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.733536 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:32:23.779268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.779239 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-bundle\") pod \"17b3ffd7-af68-482e-b934-401ed44c55b5\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " Nov 02 19:32:23.779414 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.779340 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b3ffd7-af68-482e-b934-401ed44c55b5-kube-api-access-h69p8\" (UniqueName: \"kubernetes.io/projected/17b3ffd7-af68-482e-b934-401ed44c55b5-kube-api-access-h69p8\") pod \"17b3ffd7-af68-482e-b934-401ed44c55b5\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " Nov 02 19:32:23.779414 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.779364 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-util\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-util\") pod \"17b3ffd7-af68-482e-b934-401ed44c55b5\" (UID: \"17b3ffd7-af68-482e-b934-401ed44c55b5\") " Nov 02 19:32:23.780021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.779998 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-bundle" pod "17b3ffd7-af68-482e-b934-401ed44c55b5" (UID: "17b3ffd7-af68-482e-b934-401ed44c55b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:23.781398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.781369 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b3ffd7-af68-482e-b934-401ed44c55b5-kube-api-access-h69p8" pod "17b3ffd7-af68-482e-b934-401ed44c55b5" (UID: "17b3ffd7-af68-482e-b934-401ed44c55b5"). InnerVolumeSpecName "kube-api-access-h69p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:23.786505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.786481 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-util" pod "17b3ffd7-af68-482e-b934-401ed44c55b5" (UID: "17b3ffd7-af68-482e-b934-401ed44c55b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:23.880386 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880287 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-util\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-util\") pod \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " Nov 02 19:32:23.880562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880388 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-kube-api-access-lfn2g\" (UniqueName: \"kubernetes.io/projected/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-kube-api-access-lfn2g\") pod \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " Nov 02 19:32:23.880562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880420 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95677672-f5dd-49d1-8746-e4aa52eb6d8c-kube-api-access-7lssb\" (UniqueName: \"kubernetes.io/projected/95677672-f5dd-49d1-8746-e4aa52eb6d8c-kube-api-access-7lssb\") pod \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " Nov 02 19:32:23.880562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880463 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-util\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-util\") pod \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " Nov 02 19:32:23.880562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880505 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-bundle\") pod \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\" (UID: \"95677672-f5dd-49d1-8746-e4aa52eb6d8c\") " Nov 02 19:32:23.880784 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880613 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-bundle\") pod \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\" (UID: \"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a\") " Nov 02 19:32:23.880918 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880894 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-bundle" pod "95677672-f5dd-49d1-8746-e4aa52eb6d8c" (UID: "95677672-f5dd-49d1-8746-e4aa52eb6d8c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:23.880988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880910 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69p8\" (UniqueName: \"kubernetes.io/projected/17b3ffd7-af68-482e-b934-401ed44c55b5-kube-api-access-h69p8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.880988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880933 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.880988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.880946 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b3ffd7-af68-482e-b934-401ed44c55b5-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.881280 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.881260 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-bundle" pod "b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" (UID: "b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:23.882720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.882697 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-kube-api-access-lfn2g" pod "b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" (UID: "b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a"). InnerVolumeSpecName "kube-api-access-lfn2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:23.882880 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.882864 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95677672-f5dd-49d1-8746-e4aa52eb6d8c-kube-api-access-7lssb" pod "95677672-f5dd-49d1-8746-e4aa52eb6d8c" (UID: "95677672-f5dd-49d1-8746-e4aa52eb6d8c"). InnerVolumeSpecName "kube-api-access-7lssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:23.885726 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.885701 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-util" pod "b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" (UID: "b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:23.886790 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.886768 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-util" pod "95677672-f5dd-49d1-8746-e4aa52eb6d8c" (UID: "95677672-f5dd-49d1-8746-e4aa52eb6d8c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:23.982004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.981951 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.982004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.981997 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfn2g\" (UniqueName: \"kubernetes.io/projected/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-kube-api-access-lfn2g\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.982004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.982012 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lssb\" (UniqueName: \"kubernetes.io/projected/95677672-f5dd-49d1-8746-e4aa52eb6d8c-kube-api-access-7lssb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.982281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.982025 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.982281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.982039 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95677672-f5dd-49d1-8746-e4aa52eb6d8c-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:23.982281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:23.982050 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:24.552800 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.552769 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" Nov 02 19:32:24.552800 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.552771 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449lznn" event={"ID":"95677672-f5dd-49d1-8746-e4aa52eb6d8c","Type":"ContainerDied","Data":"6817aab1a39c17bde4882eff2607973c19f49aa3829497bb83aa7e749991a706"} Nov 02 19:32:24.553252 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.552840 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6817aab1a39c17bde4882eff2607973c19f49aa3829497bb83aa7e749991a706" Nov 02 19:32:24.555537 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.555518 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" Nov 02 19:32:24.555685 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.555550 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn2bpv" event={"ID":"17b3ffd7-af68-482e-b934-401ed44c55b5","Type":"ContainerDied","Data":"aff824c149fd26849fc7fff8b651cc23e1f5867d9d869d7f7676d5e064275b26"} Nov 02 19:32:24.555685 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.555592 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff824c149fd26849fc7fff8b651cc23e1f5867d9d869d7f7676d5e064275b26" Nov 02 19:32:24.556814 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.556798 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:32:24.557622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.557599 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" event={"ID":"b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a","Type":"ContainerDied","Data":"89d823538b994d3d46a6e224f2410ed7374ee584dd10040bd4551390ae309987"} Nov 02 19:32:24.557705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.557636 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89d823538b994d3d46a6e224f2410ed7374ee584dd10040bd4551390ae309987" Nov 02 19:32:24.557705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:24.557687 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn" Nov 02 19:32:26.608451 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608421 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b"] Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608718 4966 cpu_manager.go:414] "containerMap: removing container" podUID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608730 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608738 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608743 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608754 4966 cpu_manager.go:414] "containerMap: removing container" podUID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608758 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608766 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608771 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608778 4966 cpu_manager.go:414] "containerMap: removing container" podUID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608782 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608789 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608793 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608799 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608803 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608812 4966 cpu_manager.go:414] "containerMap: removing container" podUID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608816 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608828 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608832 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608838 4966 cpu_manager.go:414] "containerMap: removing container" podUID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608842 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608847 4966 cpu_manager.go:414] "containerMap: removing container" podUID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608851 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608860 4966 cpu_manager.go:414] "containerMap: removing container" podUID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608864 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608871 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608877 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608883 4966 cpu_manager.go:414] "containerMap: removing container" podUID="af1f44a6-c60c-471b-a708-a616851b1059" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608888 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1f44a6-c60c-471b-a708-a616851b1059" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608893 4966 cpu_manager.go:414] "containerMap: removing container" podUID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608897 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerName="util" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608902 4966 cpu_manager.go:414] "containerMap: removing container" podUID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608906 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608912 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608916 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerName="extract" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608921 4966 cpu_manager.go:414] "containerMap: removing container" podUID="af1f44a6-c60c-471b-a708-a616851b1059" containerName="pull" Nov 02 19:32:26.608866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608925 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1f44a6-c60c-471b-a708-a616851b1059" containerName="pull" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608931 4966 cpu_manager.go:414] "containerMap: removing container" podUID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608943 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608950 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608954 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608962 4966 cpu_manager.go:414] "containerMap: removing container" podUID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608966 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608971 4966 cpu_manager.go:414] "containerMap: removing container" podUID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608975 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerName="util" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608981 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerName="pull" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608984 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerName="pull" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608990 4966 cpu_manager.go:414] "containerMap: removing container" podUID="af1f44a6-c60c-471b-a708-a616851b1059" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.608994 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1f44a6-c60c-471b-a708-a616851b1059" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609063 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d7c87b-1ede-4b8e-8b66-523d832063f8" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609069 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7798e0a7-c083-4e6e-8e24-194000cfa12b" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609080 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1f44a6-c60c-471b-a708-a616851b1059" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609088 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3c9855-9a5a-4c7e-9f7f-343e276f7e74" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609095 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b3ffd7-af68-482e-b934-401ed44c55b5" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609101 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ae49de-5511-4da0-b8d1-0015258bc8d9" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609106 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="95677672-f5dd-49d1-8746-e4aa52eb6d8c" containerName="extract" Nov 02 19:32:26.610061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.609111 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a" containerName="extract" Nov 02 19:32:26.612150 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.612134 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.663216 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.663185 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 19:32:26.663393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.663272 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 19:32:26.663393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.663335 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 19:32:26.663595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.663576 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 19:32:26.702749 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.702711 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.702945 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.702752 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92bd590d-de99-4d29-8c2b-3ffabf0c661d-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.702945 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.702839 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtd7\" (UniqueName: \"kubernetes.io/projected/92bd590d-de99-4d29-8c2b-3ffabf0c661d-kube-api-access-vrtd7\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.702945 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.702939 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92bd590d-de99-4d29-8c2b-3ffabf0c661d-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.703113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.702993 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.703113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.703042 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.709362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.709339 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 19:32:26.709362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.709339 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lqjs8\"" Nov 02 19:32:26.722559 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.722530 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b"] Nov 02 19:32:26.804218 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.804180 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92bd590d-de99-4d29-8c2b-3ffabf0c661d-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.804406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.804237 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtd7\" (UniqueName: \"kubernetes.io/projected/92bd590d-de99-4d29-8c2b-3ffabf0c661d-kube-api-access-vrtd7\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.804406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.804269 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92bd590d-de99-4d29-8c2b-3ffabf0c661d-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.804406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.804317 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.804406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.804355 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.804616 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.804408 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.804702 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.804680 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92bd590d-de99-4d29-8c2b-3ffabf0c661d-manifests\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.806793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.806762 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92bd590d-de99-4d29-8c2b-3ffabf0c661d-tmp\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.806914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.806815 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.806914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.806901 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-metrics-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.807017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.806979 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92bd590d-de99-4d29-8c2b-3ffabf0c661d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.859389 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.859289 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtd7\" (UniqueName: \"kubernetes.io/projected/92bd590d-de99-4d29-8c2b-3ffabf0c661d-kube-api-access-vrtd7\") pod \"opendatahub-operator-controller-manager-5985f94fb7-l5z8b\" (UID: \"92bd590d-de99-4d29-8c2b-3ffabf0c661d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:26.926378 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:26.926343 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:27.401898 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:27.401844 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b"] Nov 02 19:32:27.404145 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:32:27.404115 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bd590d_de99_4d29_8c2b_3ffabf0c661d.slice/crio-b85bd3999f00d245cdef4f5bd82ecc192bde9be869a6381c22a6a452c917f382 WatchSource:0}: Error finding container b85bd3999f00d245cdef4f5bd82ecc192bde9be869a6381c22a6a452c917f382: Status 404 returned error can't find the container with id b85bd3999f00d245cdef4f5bd82ecc192bde9be869a6381c22a6a452c917f382 Nov 02 19:32:27.572947 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:27.572912 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" event={"ID":"92bd590d-de99-4d29-8c2b-3ffabf0c661d","Type":"ContainerStarted","Data":"b85bd3999f00d245cdef4f5bd82ecc192bde9be869a6381c22a6a452c917f382"} Nov 02 19:32:31.594919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:31.594884 4966 generic.go:356] "Generic (PLEG): container finished" podID="92bd590d-de99-4d29-8c2b-3ffabf0c661d" containerID="67e335e64feb918a290058ec00c35dbe702d7fcbc6aafeafab64511a854d6a31" exitCode=0 Nov 02 19:32:31.595313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:31.594971 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" event={"ID":"92bd590d-de99-4d29-8c2b-3ffabf0c661d","Type":"ContainerDied","Data":"67e335e64feb918a290058ec00c35dbe702d7fcbc6aafeafab64511a854d6a31"} Nov 02 19:32:32.052211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.052067 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk"] Nov 02 19:32:32.075584 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.055537 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.105061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.105030 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 19:32:32.105201 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.105072 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 19:32:32.105201 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.105184 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8fnkt\"" Nov 02 19:32:32.110928 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.110898 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk"] Nov 02 19:32:32.150633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.150598 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45s75\" (UniqueName: \"kubernetes.io/projected/d39c5542-9d1e-4eab-a3be-2473a3910efb-kube-api-access-45s75\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.150779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.150726 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.150779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.150752 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.252156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.252116 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45s75\" (UniqueName: \"kubernetes.io/projected/d39c5542-9d1e-4eab-a3be-2473a3910efb-kube-api-access-45s75\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.252367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.252179 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.252367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.252197 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.252578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.252558 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.252646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.252621 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.305667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.305590 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45s75\" (UniqueName: \"kubernetes.io/projected/d39c5542-9d1e-4eab-a3be-2473a3910efb-kube-api-access-45s75\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.373096 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.373056 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:32.735864 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:32.735838 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk"] Nov 02 19:32:32.738023 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:32:32.737992 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39c5542_9d1e_4eab_a3be_2473a3910efb.slice/crio-35536632ce64515e591678d79d3011eddfb081b912a41eea800b0252981e3271 WatchSource:0}: Error finding container 35536632ce64515e591678d79d3011eddfb081b912a41eea800b0252981e3271: Status 404 returned error can't find the container with id 35536632ce64515e591678d79d3011eddfb081b912a41eea800b0252981e3271 Nov 02 19:32:33.606757 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:33.606719 4966 generic.go:356] "Generic (PLEG): container finished" podID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerID="ddb3757d93a57db94f71df2f13438f17cd8f20ee5e831e33af8750aa750670a2" exitCode=0 Nov 02 19:32:33.607104 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:33.606806 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" event={"ID":"d39c5542-9d1e-4eab-a3be-2473a3910efb","Type":"ContainerDied","Data":"ddb3757d93a57db94f71df2f13438f17cd8f20ee5e831e33af8750aa750670a2"} Nov 02 19:32:33.607104 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:33.606860 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" event={"ID":"d39c5542-9d1e-4eab-a3be-2473a3910efb","Type":"ContainerStarted","Data":"35536632ce64515e591678d79d3011eddfb081b912a41eea800b0252981e3271"} Nov 02 19:32:33.609497 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:33.609469 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" event={"ID":"92bd590d-de99-4d29-8c2b-3ffabf0c661d","Type":"ContainerStarted","Data":"b6e676864251a0f5b373bd21d4c7b81546f8bee699d147c133660348d66eed0b"} Nov 02 19:32:33.609700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:33.609625 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:33.935903 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:33.935769 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" podStartSLOduration=3.110123653 podStartE2EDuration="7.935748022s" totalImagesPullingTime="4.402686805s" totalInitContainerRuntime="422.937564ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:33.931764799 +0000 UTC m=+259.880420639" watchObservedRunningTime="2026-11-02 19:32:33.935748022 +0000 UTC m=+259.884403863" Nov 02 19:32:34.567709 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.567676 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk"] Nov 02 19:32:34.571064 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.571047 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" Nov 02 19:32:34.615909 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.615876 4966 generic.go:356] "Generic (PLEG): container finished" podID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerID="1cc5a38c9c861c46325dfd25044063c49e8dab249a07fc884929518174a25376" exitCode=0 Nov 02 19:32:34.616053 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.615963 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" event={"ID":"d39c5542-9d1e-4eab-a3be-2473a3910efb","Type":"ContainerDied","Data":"1cc5a38c9c861c46325dfd25044063c49e8dab249a07fc884929518174a25376"} Nov 02 19:32:34.620124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.620105 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 19:32:34.620226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.620149 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 19:32:34.627608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.627590 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk"] Nov 02 19:32:34.672687 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.672657 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdvl\" (UniqueName: \"kubernetes.io/projected/3e87633c-6b24-4c24-819d-ee74ce25ede3-kube-api-access-6xdvl\") pod \"limitador-operator-controller-manager-5778bb4449-qv4jk\" (UID: \"3e87633c-6b24-4c24-819d-ee74ce25ede3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" Nov 02 19:32:34.711272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.711235 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vfpcs\"" Nov 02 19:32:34.773464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.773428 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdvl\" (UniqueName: \"kubernetes.io/projected/3e87633c-6b24-4c24-819d-ee74ce25ede3-kube-api-access-6xdvl\") pod \"limitador-operator-controller-manager-5778bb4449-qv4jk\" (UID: \"3e87633c-6b24-4c24-819d-ee74ce25ede3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" Nov 02 19:32:34.827656 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.827588 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdvl\" (UniqueName: \"kubernetes.io/projected/3e87633c-6b24-4c24-819d-ee74ce25ede3-kube-api-access-6xdvl\") pod \"limitador-operator-controller-manager-5778bb4449-qv4jk\" (UID: \"3e87633c-6b24-4c24-819d-ee74ce25ede3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" Nov 02 19:32:34.883502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:34.883470 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" Nov 02 19:32:35.248382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:35.248355 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk"] Nov 02 19:32:35.313267 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:32:35.313230 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e87633c_6b24_4c24_819d_ee74ce25ede3.slice/crio-90a599fb820fbd4232376ffccf0b9c9785b3efe209d79e1ac10104f3d4615f1e WatchSource:0}: Error finding container 90a599fb820fbd4232376ffccf0b9c9785b3efe209d79e1ac10104f3d4615f1e: Status 404 returned error can't find the container with id 90a599fb820fbd4232376ffccf0b9c9785b3efe209d79e1ac10104f3d4615f1e Nov 02 19:32:35.621981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:35.621892 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" event={"ID":"3e87633c-6b24-4c24-819d-ee74ce25ede3","Type":"ContainerStarted","Data":"90a599fb820fbd4232376ffccf0b9c9785b3efe209d79e1ac10104f3d4615f1e"} Nov 02 19:32:35.624023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:35.623994 4966 generic.go:356] "Generic (PLEG): container finished" podID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerID="26adc6cac092d22b69bfbd5ff41e53c7145dc2c0d3927b6e137d09f5af667505" exitCode=0 Nov 02 19:32:35.624146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:35.624039 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" event={"ID":"d39c5542-9d1e-4eab-a3be-2473a3910efb","Type":"ContainerDied","Data":"26adc6cac092d22b69bfbd5ff41e53c7145dc2c0d3927b6e137d09f5af667505"} Nov 02 19:32:36.755013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.754985 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:36.891585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.891511 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-bundle\") pod \"d39c5542-9d1e-4eab-a3be-2473a3910efb\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " Nov 02 19:32:36.891731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.891597 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39c5542-9d1e-4eab-a3be-2473a3910efb-kube-api-access-45s75\" (UniqueName: \"kubernetes.io/projected/d39c5542-9d1e-4eab-a3be-2473a3910efb-kube-api-access-45s75\") pod \"d39c5542-9d1e-4eab-a3be-2473a3910efb\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " Nov 02 19:32:36.891731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.891628 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-util\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-util\") pod \"d39c5542-9d1e-4eab-a3be-2473a3910efb\" (UID: \"d39c5542-9d1e-4eab-a3be-2473a3910efb\") " Nov 02 19:32:36.892256 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.892226 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-bundle" pod "d39c5542-9d1e-4eab-a3be-2473a3910efb" (UID: "d39c5542-9d1e-4eab-a3be-2473a3910efb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:36.893714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.893684 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39c5542-9d1e-4eab-a3be-2473a3910efb-kube-api-access-45s75" pod "d39c5542-9d1e-4eab-a3be-2473a3910efb" (UID: "d39c5542-9d1e-4eab-a3be-2473a3910efb"). InnerVolumeSpecName "kube-api-access-45s75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:32:36.897833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.897789 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-util" pod "d39c5542-9d1e-4eab-a3be-2473a3910efb" (UID: "d39c5542-9d1e-4eab-a3be-2473a3910efb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:32:36.993041 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.992996 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45s75\" (UniqueName: \"kubernetes.io/projected/d39c5542-9d1e-4eab-a3be-2473a3910efb-kube-api-access-45s75\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:36.993041 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.993036 4966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-util\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:36.993041 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:36.993047 4966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d39c5542-9d1e-4eab-a3be-2473a3910efb-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:32:37.634639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:37.634602 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" event={"ID":"d39c5542-9d1e-4eab-a3be-2473a3910efb","Type":"ContainerDied","Data":"35536632ce64515e591678d79d3011eddfb081b912a41eea800b0252981e3271"} Nov 02 19:32:37.634803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:37.634654 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35536632ce64515e591678d79d3011eddfb081b912a41eea800b0252981e3271" Nov 02 19:32:37.634803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:37.634611 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmp8xk" Nov 02 19:32:39.646067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:39.646031 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" event={"ID":"3e87633c-6b24-4c24-819d-ee74ce25ede3","Type":"ContainerStarted","Data":"8433f0a32cf4000d38ce7805424bfffbdcafca870b2ca96f7c388f21a6cfcce0"} Nov 02 19:32:39.646441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:39.646116 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" Nov 02 19:32:39.752284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:39.752233 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" podStartSLOduration=1.616721737 podStartE2EDuration="5.752218848s" totalImagesPullingTime="4.135497111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:39.750256056 +0000 UTC m=+265.698911899" watchObservedRunningTime="2026-11-02 19:32:39.752218848 +0000 UTC m=+265.700874686" Nov 02 19:32:42.538420 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538386 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn"] Nov 02 19:32:42.538796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538696 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerName="pull" Nov 02 19:32:42.538796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538707 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerName="pull" Nov 02 19:32:42.538796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538732 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerName="extract" Nov 02 19:32:42.538796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538736 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerName="extract" Nov 02 19:32:42.538796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538744 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerName="util" Nov 02 19:32:42.538796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538749 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerName="util" Nov 02 19:32:42.539005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.538803 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39c5542-9d1e-4eab-a3be-2473a3910efb" containerName="extract" Nov 02 19:32:42.541839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.541821 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:42.595282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.595258 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4gc5k\"" Nov 02 19:32:42.637361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.637331 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94176584-5218-4e15-9061-5e0ecd6867fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vkkfn\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:42.637479 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.637440 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6s4w\" (UniqueName: \"kubernetes.io/projected/94176584-5218-4e15-9061-5e0ecd6867fe-kube-api-access-j6s4w\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vkkfn\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:42.641744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.641723 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn"] Nov 02 19:32:42.738696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.738661 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6s4w\" (UniqueName: \"kubernetes.io/projected/94176584-5218-4e15-9061-5e0ecd6867fe-kube-api-access-j6s4w\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vkkfn\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:42.738879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.738725 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94176584-5218-4e15-9061-5e0ecd6867fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vkkfn\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:42.739143 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.739126 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94176584-5218-4e15-9061-5e0ecd6867fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vkkfn\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:42.792043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.791972 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6s4w\" (UniqueName: \"kubernetes.io/projected/94176584-5218-4e15-9061-5e0ecd6867fe-kube-api-access-j6s4w\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vkkfn\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:42.854435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:42.854404 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:43.270059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:43.269634 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn"] Nov 02 19:32:43.271622 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:32:43.271598 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94176584_5218_4e15_9061_5e0ecd6867fe.slice/crio-31be1b84c4c16dc183379d2a7d2654ef3d2e3ae82245e7b59eced03b796b8762 WatchSource:0}: Error finding container 31be1b84c4c16dc183379d2a7d2654ef3d2e3ae82245e7b59eced03b796b8762: Status 404 returned error can't find the container with id 31be1b84c4c16dc183379d2a7d2654ef3d2e3ae82245e7b59eced03b796b8762 Nov 02 19:32:43.665204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:43.665121 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" event={"ID":"94176584-5218-4e15-9061-5e0ecd6867fe","Type":"ContainerStarted","Data":"31be1b84c4c16dc183379d2a7d2654ef3d2e3ae82245e7b59eced03b796b8762"} Nov 02 19:32:44.618438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:44.618403 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5985f94fb7-l5z8b" Nov 02 19:32:44.752579 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:32:44.752534 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:32:47.035908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.035872 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j"] Nov 02 19:32:47.061985 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.061956 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" Nov 02 19:32:47.077613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.077581 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkz6f\" (UniqueName: \"kubernetes.io/projected/5771e83c-8ce8-45d0-94d9-c9ef32ece870-kube-api-access-vkz6f\") pod \"dns-operator-controller-manager-8557cbdb84-66g5j\" (UID: \"5771e83c-8ce8-45d0-94d9-c9ef32ece870\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" Nov 02 19:32:47.113634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.113586 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 19:32:47.161347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.159064 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2jwnc\"" Nov 02 19:32:47.161347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.161249 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j"] Nov 02 19:32:47.179020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.178988 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkz6f\" (UniqueName: \"kubernetes.io/projected/5771e83c-8ce8-45d0-94d9-c9ef32ece870-kube-api-access-vkz6f\") pod \"dns-operator-controller-manager-8557cbdb84-66g5j\" (UID: \"5771e83c-8ce8-45d0-94d9-c9ef32ece870\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" Nov 02 19:32:47.235627 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.235590 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkz6f\" (UniqueName: \"kubernetes.io/projected/5771e83c-8ce8-45d0-94d9-c9ef32ece870-kube-api-access-vkz6f\") pod \"dns-operator-controller-manager-8557cbdb84-66g5j\" (UID: \"5771e83c-8ce8-45d0-94d9-c9ef32ece870\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" Nov 02 19:32:47.395143 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.395072 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" Nov 02 19:32:47.777650 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:47.777629 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j"] Nov 02 19:32:47.780518 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:32:47.780482 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5771e83c_8ce8_45d0_94d9_c9ef32ece870.slice/crio-82ab6cddf53939f443127a309a2ab86b99320bfc161dc70651dea712ac7ca111 WatchSource:0}: Error finding container 82ab6cddf53939f443127a309a2ab86b99320bfc161dc70651dea712ac7ca111: Status 404 returned error can't find the container with id 82ab6cddf53939f443127a309a2ab86b99320bfc161dc70651dea712ac7ca111 Nov 02 19:32:48.694817 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:48.694770 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" event={"ID":"5771e83c-8ce8-45d0-94d9-c9ef32ece870","Type":"ContainerStarted","Data":"82ab6cddf53939f443127a309a2ab86b99320bfc161dc70651dea712ac7ca111"} Nov 02 19:32:49.701855 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:49.701815 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" event={"ID":"94176584-5218-4e15-9061-5e0ecd6867fe","Type":"ContainerStarted","Data":"d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753"} Nov 02 19:32:49.702343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:49.701943 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:32:49.906655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:49.906546 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" podStartSLOduration=1.7027555570000001 podStartE2EDuration="7.906531024s" totalImagesPullingTime="6.203775467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:49.904538405 +0000 UTC m=+275.853194246" watchObservedRunningTime="2026-11-02 19:32:49.906531024 +0000 UTC m=+275.855186862" Nov 02 19:32:50.653687 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:50.653656 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qv4jk" Nov 02 19:32:52.718659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:52.718620 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" event={"ID":"5771e83c-8ce8-45d0-94d9-c9ef32ece870","Type":"ContainerStarted","Data":"b1b8f46053a60248452f8390b4bed611b0eb719828b126cae4f4788e2a7bd62e"} Nov 02 19:32:52.719063 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:52.718689 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" Nov 02 19:32:52.831130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:52.831075 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" podStartSLOduration=2.712262374 podStartE2EDuration="6.831061291s" totalImagesPullingTime="4.118798917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:52.82878646 +0000 UTC m=+278.777442299" watchObservedRunningTime="2026-11-02 19:32:52.831061291 +0000 UTC m=+278.779717131" Nov 02 19:32:54.831184 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.831149 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n"] Nov 02 19:32:54.834911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.834884 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:54.885528 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.885484 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8vqr7\"" Nov 02 19:32:54.885528 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.885501 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:32:54.931413 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.931384 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 19:32:54.933465 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.933445 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n"] Nov 02 19:32:54.940946 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.940925 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d330c46-fb7b-4284-9d7d-bb12eaab8ca3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ld64n\" (UID: \"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:54.941013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:54.940995 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49jj\" (UniqueName: \"kubernetes.io/projected/6d330c46-fb7b-4284-9d7d-bb12eaab8ca3-kube-api-access-h49jj\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ld64n\" (UID: \"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:55.041868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:55.041837 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49jj\" (UniqueName: \"kubernetes.io/projected/6d330c46-fb7b-4284-9d7d-bb12eaab8ca3-kube-api-access-h49jj\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ld64n\" (UID: \"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:55.042025 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:55.041888 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d330c46-fb7b-4284-9d7d-bb12eaab8ca3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ld64n\" (UID: \"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:55.042340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:55.042317 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d330c46-fb7b-4284-9d7d-bb12eaab8ca3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ld64n\" (UID: \"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:55.105680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:55.105609 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49jj\" (UniqueName: \"kubernetes.io/projected/6d330c46-fb7b-4284-9d7d-bb12eaab8ca3-kube-api-access-h49jj\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ld64n\" (UID: \"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:55.146050 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:55.146019 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" Nov 02 19:32:55.609482 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:55.609345 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n"] Nov 02 19:32:55.612110 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:32:55.612078 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d330c46_fb7b_4284_9d7d_bb12eaab8ca3.slice/crio-31b485785f75ca60d69ecc93e673b3310ebf7414fc6b8e2a44c86aadf2280faa WatchSource:0}: Error finding container 31b485785f75ca60d69ecc93e673b3310ebf7414fc6b8e2a44c86aadf2280faa: Status 404 returned error can't find the container with id 31b485785f75ca60d69ecc93e673b3310ebf7414fc6b8e2a44c86aadf2280faa Nov 02 19:32:55.734980 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:55.734946 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" event={"ID":"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3","Type":"ContainerStarted","Data":"31b485785f75ca60d69ecc93e673b3310ebf7414fc6b8e2a44c86aadf2280faa"} Nov 02 19:32:59.759043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:59.758997 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" event={"ID":"6d330c46-fb7b-4284-9d7d-bb12eaab8ca3","Type":"ContainerStarted","Data":"646d6819d1c6af3df71d7b9cb91ffda603ec14041b373819b2afd2ad8e4cd0c6"} Nov 02 19:32:59.962743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:32:59.962684 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ld64n" podStartSLOduration=2.641203172 podStartE2EDuration="5.962668474s" totalImagesPullingTime="3.321465302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:32:59.915110513 +0000 UTC m=+285.863766376" watchObservedRunningTime="2026-11-02 19:32:59.962668474 +0000 UTC m=+285.911324313" Nov 02 19:33:00.709253 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:00.709220 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:33:01.535718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.535679 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h"] Nov 02 19:33:01.542404 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.542378 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:01.601482 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.601439 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wgs9h\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:01.601881 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.601847 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwxm\" (UniqueName: \"kubernetes.io/projected/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-kube-api-access-kvwxm\") pod \"kuadrant-operator-controller-manager-678d87766f-wgs9h\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:01.603793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.603767 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h"] Nov 02 19:33:01.702944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.702907 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwxm\" (UniqueName: \"kubernetes.io/projected/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-kube-api-access-kvwxm\") pod \"kuadrant-operator-controller-manager-678d87766f-wgs9h\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:01.703131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.702962 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wgs9h\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:01.703421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.703401 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wgs9h\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:01.760020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.759988 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwxm\" (UniqueName: \"kubernetes.io/projected/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-kube-api-access-kvwxm\") pod \"kuadrant-operator-controller-manager-678d87766f-wgs9h\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:01.855876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:01.855800 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:02.358646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:02.358620 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h"] Nov 02 19:33:02.360057 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:33:02.360026 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482a3710_b7e5_4d30_bc84_c37fdb5cf8ed.slice/crio-510dcd86cfab67228cc88337f9ec16175a09fba32c24f90d5d237ba2ab1c1050 WatchSource:0}: Error finding container 510dcd86cfab67228cc88337f9ec16175a09fba32c24f90d5d237ba2ab1c1050: Status 404 returned error can't find the container with id 510dcd86cfab67228cc88337f9ec16175a09fba32c24f90d5d237ba2ab1c1050 Nov 02 19:33:02.775797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:02.775759 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" event={"ID":"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed","Type":"ContainerStarted","Data":"ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10"} Nov 02 19:33:02.776247 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:02.775819 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" event={"ID":"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed","Type":"ContainerStarted","Data":"510dcd86cfab67228cc88337f9ec16175a09fba32c24f90d5d237ba2ab1c1050"} Nov 02 19:33:02.776247 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:02.775862 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:02.891377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:02.891329 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" podStartSLOduration=1.891291466 podStartE2EDuration="1.891291466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:02.889925038 +0000 UTC m=+288.838580876" watchObservedRunningTime="2026-11-02 19:33:02.891291466 +0000 UTC m=+288.839947305" Nov 02 19:33:03.726428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:03.726398 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-66g5j" Nov 02 19:33:05.163463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.163424 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw"] Nov 02 19:33:05.167439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.167418 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.216518 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.216478 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qw4dx\"" Nov 02 19:33:05.238922 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.238884 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.238922 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.238916 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcqf\" (UniqueName: \"kubernetes.io/projected/9ec0732e-a572-4c2d-bc02-708dc2480f1c-kube-api-access-qqcqf\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.239174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.238937 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.239174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.239001 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.239174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.239038 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.239174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.239116 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.239174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.239149 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.239464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.239217 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.239464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.239246 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.263907 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.263874 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw"] Nov 02 19:33:05.340432 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340389 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340509 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340547 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340589 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340621 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340642 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcqf\" (UniqueName: \"kubernetes.io/projected/9ec0732e-a572-4c2d-bc02-708dc2480f1c-kube-api-access-qqcqf\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340671 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340722 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340747 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.340862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340824 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.341119 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.340890 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.341119 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.341070 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.341213 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.341126 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.341476 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.341447 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.343009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.342984 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.343429 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.343409 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.396241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.396202 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec0732e-a572-4c2d-bc02-708dc2480f1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.396478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.396453 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcqf\" (UniqueName: \"kubernetes.io/projected/9ec0732e-a572-4c2d-bc02-708dc2480f1c-kube-api-access-qqcqf\") pod \"data-science-gateway-data-science-gateway-class-6b748b65854hbcw\" (UID: \"9ec0732e-a572-4c2d-bc02-708dc2480f1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.481041 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.481002 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:05.660780 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.660737 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kjstl"] Nov 02 19:33:05.667670 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.667645 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kjstl" Nov 02 19:33:05.717468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.716750 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gd7jk\"" Nov 02 19:33:05.718982 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.718950 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kjstl"] Nov 02 19:33:05.744545 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.744479 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpb86\" (UniqueName: \"kubernetes.io/projected/7b849a5a-ef8a-4341-a14a-be857c34203f-kube-api-access-fpb86\") pod \"authorino-57f7f8659f-kjstl\" (UID: \"7b849a5a-ef8a-4341-a14a-be857c34203f\") " pod="kuadrant-system/authorino-57f7f8659f-kjstl" Nov 02 19:33:05.845134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.845087 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpb86\" (UniqueName: \"kubernetes.io/projected/7b849a5a-ef8a-4341-a14a-be857c34203f-kube-api-access-fpb86\") pod \"authorino-57f7f8659f-kjstl\" (UID: \"7b849a5a-ef8a-4341-a14a-be857c34203f\") " pod="kuadrant-system/authorino-57f7f8659f-kjstl" Nov 02 19:33:05.898753 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.898718 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpb86\" (UniqueName: \"kubernetes.io/projected/7b849a5a-ef8a-4341-a14a-be857c34203f-kube-api-access-fpb86\") pod \"authorino-57f7f8659f-kjstl\" (UID: \"7b849a5a-ef8a-4341-a14a-be857c34203f\") " pod="kuadrant-system/authorino-57f7f8659f-kjstl" Nov 02 19:33:05.928120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.928081 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw"] Nov 02 19:33:05.931376 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:33:05.931341 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec0732e_a572_4c2d_bc02_708dc2480f1c.slice/crio-6ab74642b19ff75d2a5012b815d6569c50900eafd594157edcb39c2065ac9527 WatchSource:0}: Error finding container 6ab74642b19ff75d2a5012b815d6569c50900eafd594157edcb39c2065ac9527: Status 404 returned error can't find the container with id 6ab74642b19ff75d2a5012b815d6569c50900eafd594157edcb39c2065ac9527 Nov 02 19:33:05.933755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.933709 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:33:05.933886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.933806 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:33:05.933886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:05.933870 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:33:06.000906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:06.000847 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kjstl" Nov 02 19:33:06.378157 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:06.378083 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kjstl"] Nov 02 19:33:06.380688 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:33:06.380649 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b849a5a_ef8a_4341_a14a_be857c34203f.slice/crio-18dcd73e89b03716cd915d0548b8978d36373c7353b5aad8ee2e2ecfa606fb12 WatchSource:0}: Error finding container 18dcd73e89b03716cd915d0548b8978d36373c7353b5aad8ee2e2ecfa606fb12: Status 404 returned error can't find the container with id 18dcd73e89b03716cd915d0548b8978d36373c7353b5aad8ee2e2ecfa606fb12 Nov 02 19:33:06.798182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:06.798144 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kjstl" event={"ID":"7b849a5a-ef8a-4341-a14a-be857c34203f","Type":"ContainerStarted","Data":"18dcd73e89b03716cd915d0548b8978d36373c7353b5aad8ee2e2ecfa606fb12"} Nov 02 19:33:06.799714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:06.799687 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" event={"ID":"9ec0732e-a572-4c2d-bc02-708dc2480f1c","Type":"ContainerStarted","Data":"d4259452c570aad606551e7091f9f08a86b2cf51ee75f42092a88a1335cb1e2a"} Nov 02 19:33:06.799841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:06.799735 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" event={"ID":"9ec0732e-a572-4c2d-bc02-708dc2480f1c","Type":"ContainerStarted","Data":"6ab74642b19ff75d2a5012b815d6569c50900eafd594157edcb39c2065ac9527"} Nov 02 19:33:06.965120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:06.965061 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" podStartSLOduration=1.965046791 podStartE2EDuration="1.965046791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:06.963104253 +0000 UTC m=+292.911760103" watchObservedRunningTime="2026-11-02 19:33:06.965046791 +0000 UTC m=+292.913702630" Nov 02 19:33:07.482075 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:07.482036 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:07.487059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:07.487031 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:07.804937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:07.804858 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:07.805797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:07.805765 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b65854hbcw" Nov 02 19:33:08.048407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.048375 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j62f7"] Nov 02 19:33:08.052935 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.052912 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" Nov 02 19:33:08.109473 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.109447 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j62f7"] Nov 02 19:33:08.168711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.168685 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqsgp\" (UniqueName: \"kubernetes.io/projected/67e8cde7-0a82-4c99-af88-772a9bf0006c-kube-api-access-rqsgp\") pod \"authorino-56c6c9ddd5-j62f7\" (UID: \"67e8cde7-0a82-4c99-af88-772a9bf0006c\") " pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" Nov 02 19:33:08.269879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.269841 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqsgp\" (UniqueName: \"kubernetes.io/projected/67e8cde7-0a82-4c99-af88-772a9bf0006c-kube-api-access-rqsgp\") pod \"authorino-56c6c9ddd5-j62f7\" (UID: \"67e8cde7-0a82-4c99-af88-772a9bf0006c\") " pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" Nov 02 19:33:08.323668 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.323582 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqsgp\" (UniqueName: \"kubernetes.io/projected/67e8cde7-0a82-4c99-af88-772a9bf0006c-kube-api-access-rqsgp\") pod \"authorino-56c6c9ddd5-j62f7\" (UID: \"67e8cde7-0a82-4c99-af88-772a9bf0006c\") " pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" Nov 02 19:33:08.365099 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.365060 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" Nov 02 19:33:08.727179 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.727150 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j62f7"] Nov 02 19:33:08.728917 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:33:08.728878 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e8cde7_0a82_4c99_af88_772a9bf0006c.slice/crio-f785d1d70d42d6a049350c46bb6ffbea2214824e3fdc0db3bb729f664625c16c WatchSource:0}: Error finding container f785d1d70d42d6a049350c46bb6ffbea2214824e3fdc0db3bb729f664625c16c: Status 404 returned error can't find the container with id f785d1d70d42d6a049350c46bb6ffbea2214824e3fdc0db3bb729f664625c16c Nov 02 19:33:08.811872 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:08.811841 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" event={"ID":"67e8cde7-0a82-4c99-af88-772a9bf0006c","Type":"ContainerStarted","Data":"f785d1d70d42d6a049350c46bb6ffbea2214824e3fdc0db3bb729f664625c16c"} Nov 02 19:33:09.817598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:09.817515 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kjstl" event={"ID":"7b849a5a-ef8a-4341-a14a-be857c34203f","Type":"ContainerStarted","Data":"6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb"} Nov 02 19:33:09.922417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:09.922313 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-kjstl" podStartSLOduration=1.793206702 podStartE2EDuration="4.922281508s" totalImagesPullingTime="3.129074806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:09.921028297 +0000 UTC m=+295.869684136" watchObservedRunningTime="2026-11-02 19:33:09.922281508 +0000 UTC m=+295.870937348" Nov 02 19:33:10.823867 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:10.823829 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" event={"ID":"67e8cde7-0a82-4c99-af88-772a9bf0006c","Type":"ContainerStarted","Data":"16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c"} Nov 02 19:33:10.935272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:10.935222 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" podStartSLOduration=1.584855753 podStartE2EDuration="2.935209199s" totalImagesPullingTime="1.350353446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:10.934229347 +0000 UTC m=+296.882885186" watchObservedRunningTime="2026-11-02 19:33:10.935209199 +0000 UTC m=+296.883865037" Nov 02 19:33:11.020853 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:11.020824 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kjstl"] Nov 02 19:33:11.829102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:11.829056 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-kjstl" podUID="7b849a5a-ef8a-4341-a14a-be857c34203f" containerName="authorino" containerID="cri-o://6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb" gracePeriod=30 Nov 02 19:33:12.081076 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.081022 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kjstl" Nov 02 19:33:12.208128 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.208101 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b849a5a-ef8a-4341-a14a-be857c34203f-kube-api-access-fpb86\" (UniqueName: \"kubernetes.io/projected/7b849a5a-ef8a-4341-a14a-be857c34203f-kube-api-access-fpb86\") pod \"7b849a5a-ef8a-4341-a14a-be857c34203f\" (UID: \"7b849a5a-ef8a-4341-a14a-be857c34203f\") " Nov 02 19:33:12.210235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.210204 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b849a5a-ef8a-4341-a14a-be857c34203f-kube-api-access-fpb86" pod "7b849a5a-ef8a-4341-a14a-be857c34203f" (UID: "7b849a5a-ef8a-4341-a14a-be857c34203f"). InnerVolumeSpecName "kube-api-access-fpb86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:33:12.309072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.309034 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpb86\" (UniqueName: \"kubernetes.io/projected/7b849a5a-ef8a-4341-a14a-be857c34203f-kube-api-access-fpb86\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:33:12.543524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.543491 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-8gt7h"] Nov 02 19:33:12.543883 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.543870 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7b849a5a-ef8a-4341-a14a-be857c34203f" containerName="authorino" Nov 02 19:33:12.543922 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.543885 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b849a5a-ef8a-4341-a14a-be857c34203f" containerName="authorino" Nov 02 19:33:12.543979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.543970 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b849a5a-ef8a-4341-a14a-be857c34203f" containerName="authorino" Nov 02 19:33:12.547725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.547707 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.601556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.601533 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8gt7h"] Nov 02 19:33:12.610699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.610674 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44fdc9c-0626-4b51-89ba-5a9c857632be-bound-sa-token\") pod \"cert-manager-597b95cb84-8gt7h\" (UID: \"d44fdc9c-0626-4b51-89ba-5a9c857632be\") " pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.610802 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.610724 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7d6m\" (UniqueName: \"kubernetes.io/projected/d44fdc9c-0626-4b51-89ba-5a9c857632be-kube-api-access-b7d6m\") pod \"cert-manager-597b95cb84-8gt7h\" (UID: \"d44fdc9c-0626-4b51-89ba-5a9c857632be\") " pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.642164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.642139 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 19:33:12.642164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.642148 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 19:33:12.642404 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.642362 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-d84cd\"" Nov 02 19:33:12.711530 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.711494 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44fdc9c-0626-4b51-89ba-5a9c857632be-bound-sa-token\") pod \"cert-manager-597b95cb84-8gt7h\" (UID: \"d44fdc9c-0626-4b51-89ba-5a9c857632be\") " pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.711711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.711549 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7d6m\" (UniqueName: \"kubernetes.io/projected/d44fdc9c-0626-4b51-89ba-5a9c857632be-kube-api-access-b7d6m\") pod \"cert-manager-597b95cb84-8gt7h\" (UID: \"d44fdc9c-0626-4b51-89ba-5a9c857632be\") " pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.764883 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.764844 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44fdc9c-0626-4b51-89ba-5a9c857632be-bound-sa-token\") pod \"cert-manager-597b95cb84-8gt7h\" (UID: \"d44fdc9c-0626-4b51-89ba-5a9c857632be\") " pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.765015 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.764976 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7d6m\" (UniqueName: \"kubernetes.io/projected/d44fdc9c-0626-4b51-89ba-5a9c857632be-kube-api-access-b7d6m\") pod \"cert-manager-597b95cb84-8gt7h\" (UID: \"d44fdc9c-0626-4b51-89ba-5a9c857632be\") " pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.836026 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.835946 4966 generic.go:356] "Generic (PLEG): container finished" podID="7b849a5a-ef8a-4341-a14a-be857c34203f" containerID="6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb" exitCode=0 Nov 02 19:33:12.836026 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.835995 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kjstl" Nov 02 19:33:12.836562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.836031 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kjstl" event={"ID":"7b849a5a-ef8a-4341-a14a-be857c34203f","Type":"ContainerDied","Data":"6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb"} Nov 02 19:33:12.836562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.836076 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kjstl" event={"ID":"7b849a5a-ef8a-4341-a14a-be857c34203f","Type":"ContainerDied","Data":"18dcd73e89b03716cd915d0548b8978d36373c7353b5aad8ee2e2ecfa606fb12"} Nov 02 19:33:12.836562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.836097 4966 scope.go:122] "RemoveContainer" containerID="6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb" Nov 02 19:33:12.850155 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.850137 4966 scope.go:122] "RemoveContainer" containerID="6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb" Nov 02 19:33:12.850418 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:33:12.850400 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb\": container with ID starting with 6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb not found: ID does not exist" containerID="6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb" Nov 02 19:33:12.850463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.850428 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb"} err="failed to get container status \"6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb\": rpc error: code = NotFound desc = could not find container \"6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb\": container with ID starting with 6d70f704a9237201ada3e9d3c00ca50c942e650ee0868a3606f6f3e3201bc1fb not found: ID does not exist" Nov 02 19:33:12.859639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.859612 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8gt7h" Nov 02 19:33:12.987646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.987613 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kjstl"] Nov 02 19:33:12.993419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:12.993390 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kjstl"] Nov 02 19:33:13.228000 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:13.225453 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8gt7h"] Nov 02 19:33:13.229802 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:33:13.229760 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd44fdc9c_0626_4b51_89ba_5a9c857632be.slice/crio-f3641fb0a067660b60d62e9aa6a6386c7fc8e819850910ddb21a9498f69d0242 WatchSource:0}: Error finding container f3641fb0a067660b60d62e9aa6a6386c7fc8e819850910ddb21a9498f69d0242: Status 404 returned error can't find the container with id f3641fb0a067660b60d62e9aa6a6386c7fc8e819850910ddb21a9498f69d0242 Nov 02 19:33:13.782960 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:13.782932 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 19:33:13.843002 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:13.842967 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8gt7h" event={"ID":"d44fdc9c-0626-4b51-89ba-5a9c857632be","Type":"ContainerStarted","Data":"f3641fb0a067660b60d62e9aa6a6386c7fc8e819850910ddb21a9498f69d0242"} Nov 02 19:33:13.988356 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:13.988320 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn"] Nov 02 19:33:13.988569 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:13.988541 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" podUID="94176584-5218-4e15-9061-5e0ecd6867fe" containerName="manager" containerID="cri-o://d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753" gracePeriod=10 Nov 02 19:33:14.182552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.182482 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b849a5a-ef8a-4341-a14a-be857c34203f" path="/var/lib/kubelet/pods/7b849a5a-ef8a-4341-a14a-be857c34203f/volumes" Nov 02 19:33:14.195199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.195167 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:33:14.201447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.201420 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:33:14.214717 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.214698 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:33:14.214994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.214978 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:33:14.640635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.640610 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:33:14.729711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.729674 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94176584-5218-4e15-9061-5e0ecd6867fe-kube-api-access-j6s4w\" (UniqueName: \"kubernetes.io/projected/94176584-5218-4e15-9061-5e0ecd6867fe-kube-api-access-j6s4w\") pod \"94176584-5218-4e15-9061-5e0ecd6867fe\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " Nov 02 19:33:14.729889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.729724 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94176584-5218-4e15-9061-5e0ecd6867fe-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94176584-5218-4e15-9061-5e0ecd6867fe-extensions-socket-volume\") pod \"94176584-5218-4e15-9061-5e0ecd6867fe\" (UID: \"94176584-5218-4e15-9061-5e0ecd6867fe\") " Nov 02 19:33:14.730224 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.730197 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94176584-5218-4e15-9061-5e0ecd6867fe-extensions-socket-volume" pod "94176584-5218-4e15-9061-5e0ecd6867fe" (UID: "94176584-5218-4e15-9061-5e0ecd6867fe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:33:14.731919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.731894 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94176584-5218-4e15-9061-5e0ecd6867fe-kube-api-access-j6s4w" pod "94176584-5218-4e15-9061-5e0ecd6867fe" (UID: "94176584-5218-4e15-9061-5e0ecd6867fe"). InnerVolumeSpecName "kube-api-access-j6s4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:33:14.751842 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:33:14.751807 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:33:14.830402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.830324 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6s4w\" (UniqueName: \"kubernetes.io/projected/94176584-5218-4e15-9061-5e0ecd6867fe-kube-api-access-j6s4w\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:33:14.830402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.830354 4966 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94176584-5218-4e15-9061-5e0ecd6867fe-extensions-socket-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:33:14.850106 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.850077 4966 generic.go:356] "Generic (PLEG): container finished" podID="94176584-5218-4e15-9061-5e0ecd6867fe" containerID="d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753" exitCode=0 Nov 02 19:33:14.850551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.850128 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" Nov 02 19:33:14.850551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.850161 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" event={"ID":"94176584-5218-4e15-9061-5e0ecd6867fe","Type":"ContainerDied","Data":"d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753"} Nov 02 19:33:14.850551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.850238 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn" event={"ID":"94176584-5218-4e15-9061-5e0ecd6867fe","Type":"ContainerDied","Data":"31be1b84c4c16dc183379d2a7d2654ef3d2e3ae82245e7b59eced03b796b8762"} Nov 02 19:33:14.850551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.850262 4966 scope.go:122] "RemoveContainer" containerID="d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753" Nov 02 19:33:14.860641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.860619 4966 scope.go:122] "RemoveContainer" containerID="d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753" Nov 02 19:33:14.860988 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:33:14.860968 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753\": container with ID starting with d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753 not found: ID does not exist" containerID="d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753" Nov 02 19:33:14.861063 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:14.861001 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753"} err="failed to get container status \"d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753\": rpc error: code = NotFound desc = could not find container \"d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753\": container with ID starting with d335fa549c144064d521b0867fa9feeb5f07a2202b4bfe48f1839fe433ba5753 not found: ID does not exist" Nov 02 19:33:15.004313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:15.004268 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn"] Nov 02 19:33:15.012150 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:15.012121 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vkkfn"] Nov 02 19:33:16.134193 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.134159 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m"] Nov 02 19:33:16.134743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.134654 4966 cpu_manager.go:414] "containerMap: removing container" podUID="94176584-5218-4e15-9061-5e0ecd6867fe" containerName="manager" Nov 02 19:33:16.134743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.134672 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94176584-5218-4e15-9061-5e0ecd6867fe" containerName="manager" Nov 02 19:33:16.134827 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.134755 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="94176584-5218-4e15-9061-5e0ecd6867fe" containerName="manager" Nov 02 19:33:16.137545 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.137524 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.183724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.183691 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94176584-5218-4e15-9061-5e0ecd6867fe" path="/var/lib/kubelet/pods/94176584-5218-4e15-9061-5e0ecd6867fe/volumes" Nov 02 19:33:16.186498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.186472 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-v7mxx\"" Nov 02 19:33:16.232540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.232260 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 19:33:16.232540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.232270 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 19:33:16.243807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.243656 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b80d8188-2872-4626-8ee7-8177b778d47d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.243807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.243707 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8hqs\" (UniqueName: \"kubernetes.io/projected/b80d8188-2872-4626-8ee7-8177b778d47d-kube-api-access-t8hqs\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.244176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.244154 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b80d8188-2872-4626-8ee7-8177b778d47d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.244260 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.244188 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b80d8188-2872-4626-8ee7-8177b778d47d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.325147 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.325111 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 19:33:16.327730 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.327697 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m"] Nov 02 19:33:16.345412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.345385 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b80d8188-2872-4626-8ee7-8177b778d47d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.345585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.345422 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b80d8188-2872-4626-8ee7-8177b778d47d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.345585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.345489 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b80d8188-2872-4626-8ee7-8177b778d47d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.345585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.345518 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8hqs\" (UniqueName: \"kubernetes.io/projected/b80d8188-2872-4626-8ee7-8177b778d47d-kube-api-access-t8hqs\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.345926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.345904 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b80d8188-2872-4626-8ee7-8177b778d47d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.346026 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.345976 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b80d8188-2872-4626-8ee7-8177b778d47d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.346547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.346526 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b80d8188-2872-4626-8ee7-8177b778d47d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.399267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.399202 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8hqs\" (UniqueName: \"kubernetes.io/projected/b80d8188-2872-4626-8ee7-8177b778d47d-kube-api-access-t8hqs\") pod \"kserve-module-controller-manager-7566cf5fc8-4sl5m\" (UID: \"b80d8188-2872-4626-8ee7-8177b778d47d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.453285 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.453243 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:16.870445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:16.870412 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m"] Nov 02 19:33:16.872880 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:33:16.872845 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80d8188_2872_4626_8ee7_8177b778d47d.slice/crio-b9656b437332543f7b3499be5e249d0d74972e3fe8237c25420e266ec26cd96e WatchSource:0}: Error finding container b9656b437332543f7b3499be5e249d0d74972e3fe8237c25420e266ec26cd96e: Status 404 returned error can't find the container with id b9656b437332543f7b3499be5e249d0d74972e3fe8237c25420e266ec26cd96e Nov 02 19:33:17.870980 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:17.870937 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8gt7h" event={"ID":"d44fdc9c-0626-4b51-89ba-5a9c857632be","Type":"ContainerStarted","Data":"085542d10dc12bd2bd4c48024caf71df50b0ee4b5e7f86022d9b69b72aa4d7db"} Nov 02 19:33:17.872266 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:17.872228 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" event={"ID":"b80d8188-2872-4626-8ee7-8177b778d47d","Type":"ContainerStarted","Data":"b9656b437332543f7b3499be5e249d0d74972e3fe8237c25420e266ec26cd96e"} Nov 02 19:33:17.979609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:17.979544 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-8gt7h" podStartSLOduration=1.960733192 podStartE2EDuration="5.979524104s" totalImagesPullingTime="4.018790912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:17.976764501 +0000 UTC m=+303.925420339" watchObservedRunningTime="2026-11-02 19:33:17.979524104 +0000 UTC m=+303.928179945" Nov 02 19:33:19.886066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:19.886027 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" event={"ID":"b80d8188-2872-4626-8ee7-8177b778d47d","Type":"ContainerStarted","Data":"1352a170e349f2c6f322251843ac029eb70491028ce157527d5237b3a7707f53"} Nov 02 19:33:19.886484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:19.886100 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:19.995170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:19.995122 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" podStartSLOduration=1.096581309 podStartE2EDuration="3.995108555s" totalImagesPullingTime="2.898527246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:33:19.993810867 +0000 UTC m=+305.942466718" watchObservedRunningTime="2026-11-02 19:33:19.995108555 +0000 UTC m=+305.943764394" Nov 02 19:33:40.893181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:40.893103 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4sl5m" Nov 02 19:33:44.750744 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:33:44.750708 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:33:59.134605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.134571 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-h2bmf"] Nov 02 19:33:59.138004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.137985 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.187285 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.187254 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 19:33:59.187520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.187338 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 19:33:59.201885 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.201853 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xb9\" (UniqueName: \"kubernetes.io/projected/f44efe28-dfb1-41e6-b989-695ac9931c06-kube-api-access-q9xb9\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.202023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.201890 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-metrics-certs\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.202023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.201968 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.232914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.232882 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 19:33:59.233063 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.232886 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gbrc7\"" Nov 02 19:33:59.235113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.235091 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-h2bmf"] Nov 02 19:33:59.303519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.303480 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xb9\" (UniqueName: \"kubernetes.io/projected/f44efe28-dfb1-41e6-b989-695ac9931c06-kube-api-access-q9xb9\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.303704 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.303531 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-metrics-certs\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.303704 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.303587 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.303802 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:33:59.303759 4966 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Nov 02 19:33:59.303842 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:33:59.303819 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert podName:f44efe28-dfb1-41e6-b989-695ac9931c06 nodeName:}" failed. No retries permitted until 2026-11-02 19:33:59.803802713 +0000 UTC m=+345.752458548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert") pod "maas-controller-579cf7c6d-h2bmf" (UID: "f44efe28-dfb1-41e6-b989-695ac9931c06") : secret "maas-controller-webhook-cert" not found Nov 02 19:33:59.306145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.306119 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-metrics-certs\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.356806 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.356774 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xb9\" (UniqueName: \"kubernetes.io/projected/f44efe28-dfb1-41e6-b989-695ac9931c06-kube-api-access-q9xb9\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.515248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.515214 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-t47p9"] Nov 02 19:33:59.518796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.518775 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.574379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.574350 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-t47p9"] Nov 02 19:33:59.607393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.607364 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.607546 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.607517 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4h7c\" (UniqueName: \"kubernetes.io/projected/999c00dd-b142-421a-a171-a7185d979eeb-kube-api-access-c4h7c\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.607629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.607574 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.708823 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.708781 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4h7c\" (UniqueName: \"kubernetes.io/projected/999c00dd-b142-421a-a171-a7185d979eeb-kube-api-access-c4h7c\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.709001 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.708833 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.709001 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.708861 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.711564 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.711537 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.711677 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.711573 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.761709 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.761677 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4h7c\" (UniqueName: \"kubernetes.io/projected/999c00dd-b142-421a-a171-a7185d979eeb-kube-api-access-c4h7c\") pod \"maas-controller-6ddc6f7dd5-t47p9\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:33:59.809865 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.809792 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.812054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.812029 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert\") pod \"maas-controller-579cf7c6d-h2bmf\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:33:59.831949 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:33:59.831931 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:34:00.051470 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:00.051432 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:34:00.198970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:00.198935 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-t47p9"] Nov 02 19:34:00.231144 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:00.231119 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999c00dd_b142_421a_a171_a7185d979eeb.slice/crio-181bc22ceaed24734ab3c96264c75015c3fd1d459d5907b576912caa20b2140e WatchSource:0}: Error finding container 181bc22ceaed24734ab3c96264c75015c3fd1d459d5907b576912caa20b2140e: Status 404 returned error can't find the container with id 181bc22ceaed24734ab3c96264c75015c3fd1d459d5907b576912caa20b2140e Nov 02 19:34:00.232578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:00.232555 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:34:00.416667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:00.416642 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-h2bmf"] Nov 02 19:34:00.418915 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:00.418880 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44efe28_dfb1_41e6_b989_695ac9931c06.slice/crio-4566c0940d01867dd0ec103175aa5793cf07d88c4263ff591302f0ecd0d9f99f WatchSource:0}: Error finding container 4566c0940d01867dd0ec103175aa5793cf07d88c4263ff591302f0ecd0d9f99f: Status 404 returned error can't find the container with id 4566c0940d01867dd0ec103175aa5793cf07d88c4263ff591302f0ecd0d9f99f Nov 02 19:34:01.109073 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:01.109038 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" event={"ID":"f44efe28-dfb1-41e6-b989-695ac9931c06","Type":"ContainerStarted","Data":"4566c0940d01867dd0ec103175aa5793cf07d88c4263ff591302f0ecd0d9f99f"} Nov 02 19:34:01.110245 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:01.110217 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" event={"ID":"999c00dd-b142-421a-a171-a7185d979eeb","Type":"ContainerStarted","Data":"181bc22ceaed24734ab3c96264c75015c3fd1d459d5907b576912caa20b2140e"} Nov 02 19:34:04.133526 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:04.133427 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" event={"ID":"f44efe28-dfb1-41e6-b989-695ac9931c06","Type":"ContainerStarted","Data":"7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3"} Nov 02 19:34:04.133971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:04.133555 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:34:04.135067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:04.135040 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" event={"ID":"999c00dd-b142-421a-a171-a7185d979eeb","Type":"ContainerStarted","Data":"7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6"} Nov 02 19:34:04.135183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:04.135131 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:34:04.241917 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:04.241827 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" podStartSLOduration=1.8586680119999999 podStartE2EDuration="5.241812437s" totalImagesPullingTime="3.383144425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:04.24098333 +0000 UTC m=+350.189639170" watchObservedRunningTime="2026-11-02 19:34:04.241812437 +0000 UTC m=+350.190468311" Nov 02 19:34:04.404234 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:04.404121 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" podStartSLOduration=1.711629646 podStartE2EDuration="5.404104934s" totalImagesPullingTime="3.692475288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:04.403537736 +0000 UTC m=+350.352193575" watchObservedRunningTime="2026-11-02 19:34:04.404104934 +0000 UTC m=+350.352760773" Nov 02 19:34:07.020049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.020012 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb"] Nov 02 19:34:07.023632 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.023613 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.079543 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.079516 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 19:34:07.079543 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.079520 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 19:34:07.079784 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.079765 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 19:34:07.086387 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.086359 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsbqg\" (UniqueName: \"kubernetes.io/projected/332785e3-c94e-47bd-8c45-8440843708b4-kube-api-access-qsbqg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.086495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.086401 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.086555 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.086534 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.086607 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.086573 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/332785e3-c94e-47bd-8c45-8440843708b4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.126166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.126123 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-c6v6l\"" Nov 02 19:34:07.128542 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.128518 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb"] Nov 02 19:34:07.187912 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.187881 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsbqg\" (UniqueName: \"kubernetes.io/projected/332785e3-c94e-47bd-8c45-8440843708b4-kube-api-access-qsbqg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.188083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.187917 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.188083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.187965 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.188083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.187987 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/332785e3-c94e-47bd-8c45-8440843708b4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.188083 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:07.188070 4966 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 19:34:07.188290 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:07.188121 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-proxy-source podName:332785e3-c94e-47bd-8c45-8440843708b4 nodeName:}" failed. No retries permitted until 2026-11-02 19:34:07.688102386 +0000 UTC m=+353.636758203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" (UID: "332785e3-c94e-47bd-8c45-8440843708b4") : configmap "usage-tenancy-proxy-source" not found Nov 02 19:34:07.188290 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:07.188119 4966 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 19:34:07.188290 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:07.188149 4966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332785e3-c94e-47bd-8c45-8440843708b4-tls-cert podName:332785e3-c94e-47bd-8c45-8440843708b4 nodeName:}" failed. No retries permitted until 2026-11-02 19:34:07.688142654 +0000 UTC m=+353.636798484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/332785e3-c94e-47bd-8c45-8440843708b4-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" (UID: "332785e3-c94e-47bd-8c45-8440843708b4") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 19:34:07.188824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.188800 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.245201 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.245163 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsbqg\" (UniqueName: \"kubernetes.io/projected/332785e3-c94e-47bd-8c45-8440843708b4-kube-api-access-qsbqg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.691652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.691615 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.691882 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.691769 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/332785e3-c94e-47bd-8c45-8440843708b4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.692450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.692427 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/332785e3-c94e-47bd-8c45-8440843708b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.694241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.694213 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/332785e3-c94e-47bd-8c45-8440843708b4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rvzdb\" (UID: \"332785e3-c94e-47bd-8c45-8440843708b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:07.936622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:07.936591 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:08.303121 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:08.303075 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb"] Nov 02 19:34:08.304788 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:08.304755 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332785e3_c94e_47bd_8c45_8440843708b4.slice/crio-86c300c8d127c755f1279376b1d400b1da32a0601735e39212b0f2bf3f3c1fc0 WatchSource:0}: Error finding container 86c300c8d127c755f1279376b1d400b1da32a0601735e39212b0f2bf3f3c1fc0: Status 404 returned error can't find the container with id 86c300c8d127c755f1279376b1d400b1da32a0601735e39212b0f2bf3f3c1fc0 Nov 02 19:34:09.163241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:09.163207 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" event={"ID":"332785e3-c94e-47bd-8c45-8440843708b4","Type":"ContainerStarted","Data":"86c300c8d127c755f1279376b1d400b1da32a0601735e39212b0f2bf3f3c1fc0"} Nov 02 19:34:14.759157 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:14.759093 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:34:15.143977 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.143813 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:34:15.144146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.144109 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 19:34:15.422521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.422439 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-h2bmf"] Nov 02 19:34:15.424735 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.424676 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" podUID="f44efe28-dfb1-41e6-b989-695ac9931c06" containerName="manager" containerID="cri-o://7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3" gracePeriod=30 Nov 02 19:34:15.724084 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.724055 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:34:15.877056 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.877013 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert\") pod \"f44efe28-dfb1-41e6-b989-695ac9931c06\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " Nov 02 19:34:15.877509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.877093 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44efe28-dfb1-41e6-b989-695ac9931c06-kube-api-access-q9xb9\" (UniqueName: \"kubernetes.io/projected/f44efe28-dfb1-41e6-b989-695ac9931c06-kube-api-access-q9xb9\") pod \"f44efe28-dfb1-41e6-b989-695ac9931c06\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " Nov 02 19:34:15.877509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.877186 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-metrics-certs\") pod \"f44efe28-dfb1-41e6-b989-695ac9931c06\" (UID: \"f44efe28-dfb1-41e6-b989-695ac9931c06\") " Nov 02 19:34:15.879994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.879933 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert" pod "f44efe28-dfb1-41e6-b989-695ac9931c06" (UID: "f44efe28-dfb1-41e6-b989-695ac9931c06"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:34:15.879994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.879948 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44efe28-dfb1-41e6-b989-695ac9931c06-kube-api-access-q9xb9" pod "f44efe28-dfb1-41e6-b989-695ac9931c06" (UID: "f44efe28-dfb1-41e6-b989-695ac9931c06"). InnerVolumeSpecName "kube-api-access-q9xb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:34:15.880412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.880385 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-metrics-certs" pod "f44efe28-dfb1-41e6-b989-695ac9931c06" (UID: "f44efe28-dfb1-41e6-b989-695ac9931c06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:34:15.978407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.978367 4966 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-webhook-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:15.978407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.978405 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9xb9\" (UniqueName: \"kubernetes.io/projected/f44efe28-dfb1-41e6-b989-695ac9931c06-kube-api-access-q9xb9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:15.978407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:15.978415 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f44efe28-dfb1-41e6-b989-695ac9931c06-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:16.218954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.218913 4966 generic.go:356] "Generic (PLEG): container finished" podID="f44efe28-dfb1-41e6-b989-695ac9931c06" containerID="7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3" exitCode=0 Nov 02 19:34:16.219127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.218966 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" Nov 02 19:34:16.219127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.218998 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" event={"ID":"f44efe28-dfb1-41e6-b989-695ac9931c06","Type":"ContainerDied","Data":"7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3"} Nov 02 19:34:16.219127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.219057 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-h2bmf" event={"ID":"f44efe28-dfb1-41e6-b989-695ac9931c06","Type":"ContainerDied","Data":"4566c0940d01867dd0ec103175aa5793cf07d88c4263ff591302f0ecd0d9f99f"} Nov 02 19:34:16.219127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.219090 4966 scope.go:122] "RemoveContainer" containerID="7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3" Nov 02 19:34:16.230432 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.230404 4966 scope.go:122] "RemoveContainer" containerID="7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3" Nov 02 19:34:16.230783 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:16.230754 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3\": container with ID starting with 7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3 not found: ID does not exist" containerID="7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3" Nov 02 19:34:16.230891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.230799 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3"} err="failed to get container status \"7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3\": rpc error: code = NotFound desc = could not find container \"7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3\": container with ID starting with 7616889d3555621d8458c5f0b72cb4801586167a1bfdf048a7990288f952e9c3 not found: ID does not exist" Nov 02 19:34:16.372885 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.372843 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-h2bmf"] Nov 02 19:34:16.378382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:16.378256 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-h2bmf"] Nov 02 19:34:18.183157 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:18.183121 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44efe28-dfb1-41e6-b989-695ac9931c06" path="/var/lib/kubelet/pods/f44efe28-dfb1-41e6-b989-695ac9931c06/volumes" Nov 02 19:34:22.257967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.257927 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" event={"ID":"332785e3-c94e-47bd-8c45-8440843708b4","Type":"ContainerStarted","Data":"f78f7438560942f9853d5716e338feec6c6f77b608d0ce3a6bf8f5c7ac034e94"} Nov 02 19:34:22.258446 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.258064 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:22.365071 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.364729 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" podStartSLOduration=3.164688041 podStartE2EDuration="16.36471133s" totalImagesPullingTime="13.200023289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:22.363575713 +0000 UTC m=+368.312231551" watchObservedRunningTime="2026-11-02 19:34:22.36471133 +0000 UTC m=+368.313367169" Nov 02 19:34:22.417273 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.417232 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pxv8z"] Nov 02 19:34:22.417664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.417648 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f44efe28-dfb1-41e6-b989-695ac9931c06" containerName="manager" Nov 02 19:34:22.417664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.417662 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44efe28-dfb1-41e6-b989-695ac9931c06" containerName="manager" Nov 02 19:34:22.417829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.417741 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44efe28-dfb1-41e6-b989-695ac9931c06" containerName="manager" Nov 02 19:34:22.421113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.421092 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.469588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.469560 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hr65c\"" Nov 02 19:34:22.469759 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.469631 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 19:34:22.475365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.475324 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pxv8z"] Nov 02 19:34:22.521656 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.521573 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz"] Nov 02 19:34:22.525463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.525436 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.536804 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.536782 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa3d028b-c7ed-4757-9161-6b55dc780ff8-cert\") pod \"kserve-controller-manager-776cc66d47-pxv8z\" (UID: \"fa3d028b-c7ed-4757-9161-6b55dc780ff8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.536908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.536811 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xv54\" (UniqueName: \"kubernetes.io/projected/fa3d028b-c7ed-4757-9161-6b55dc780ff8-kube-api-access-8xv54\") pod \"kserve-controller-manager-776cc66d47-pxv8z\" (UID: \"fa3d028b-c7ed-4757-9161-6b55dc780ff8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.579469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.579431 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz"] Nov 02 19:34:22.637471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.637438 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9j4\" (UniqueName: \"kubernetes.io/projected/4c8be9b5-e744-400a-9c31-18cd6c3b4975-kube-api-access-ks9j4\") pod \"llmisvc-controller-manager-869d5dcc5d-qdfsz\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.637471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.637473 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa3d028b-c7ed-4757-9161-6b55dc780ff8-cert\") pod \"kserve-controller-manager-776cc66d47-pxv8z\" (UID: \"fa3d028b-c7ed-4757-9161-6b55dc780ff8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.637681 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.637492 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xv54\" (UniqueName: \"kubernetes.io/projected/fa3d028b-c7ed-4757-9161-6b55dc780ff8-kube-api-access-8xv54\") pod \"kserve-controller-manager-776cc66d47-pxv8z\" (UID: \"fa3d028b-c7ed-4757-9161-6b55dc780ff8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.637681 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.637568 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8be9b5-e744-400a-9c31-18cd6c3b4975-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qdfsz\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.640020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.640000 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa3d028b-c7ed-4757-9161-6b55dc780ff8-cert\") pod \"kserve-controller-manager-776cc66d47-pxv8z\" (UID: \"fa3d028b-c7ed-4757-9161-6b55dc780ff8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.669772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.669741 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-svv44\"" Nov 02 19:34:22.669772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.669765 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 19:34:22.690399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.690370 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xv54\" (UniqueName: \"kubernetes.io/projected/fa3d028b-c7ed-4757-9161-6b55dc780ff8-kube-api-access-8xv54\") pod \"kserve-controller-manager-776cc66d47-pxv8z\" (UID: \"fa3d028b-c7ed-4757-9161-6b55dc780ff8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.734135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.734105 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:22.736037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.736012 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-fnf5k"] Nov 02 19:34:22.738715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.738689 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9j4\" (UniqueName: \"kubernetes.io/projected/4c8be9b5-e744-400a-9c31-18cd6c3b4975-kube-api-access-ks9j4\") pod \"llmisvc-controller-manager-869d5dcc5d-qdfsz\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.738822 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.738745 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8be9b5-e744-400a-9c31-18cd6c3b4975-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qdfsz\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.739514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.739499 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.741244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.741226 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8be9b5-e744-400a-9c31-18cd6c3b4975-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qdfsz\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.789082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.789052 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 19:34:22.789208 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.789191 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 19:34:22.789282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.789270 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kp562\"" Nov 02 19:34:22.795537 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.795019 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fnf5k"] Nov 02 19:34:22.803689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.803662 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9j4\" (UniqueName: \"kubernetes.io/projected/4c8be9b5-e744-400a-9c31-18cd6c3b4975-kube-api-access-ks9j4\") pod \"llmisvc-controller-manager-869d5dcc5d-qdfsz\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.838222 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.838183 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:22.840087 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.840054 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/799aca84-787c-4e88-b7f1-fea3975ec850-cert\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.840261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.840149 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkft\" (UniqueName: \"kubernetes.io/projected/799aca84-787c-4e88-b7f1-fea3975ec850-kube-api-access-jtkft\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.840381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.840326 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/799aca84-787c-4e88-b7f1-fea3975ec850-metrics-certs\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.941062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.941022 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/799aca84-787c-4e88-b7f1-fea3975ec850-metrics-certs\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.941062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.941063 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/799aca84-787c-4e88-b7f1-fea3975ec850-cert\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.941291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.941112 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkft\" (UniqueName: \"kubernetes.io/projected/799aca84-787c-4e88-b7f1-fea3975ec850-kube-api-access-jtkft\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.943575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.943549 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/799aca84-787c-4e88-b7f1-fea3975ec850-cert\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.943687 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.943552 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/799aca84-787c-4e88-b7f1-fea3975ec850-metrics-certs\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:22.994411 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:22.994383 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkft\" (UniqueName: \"kubernetes.io/projected/799aca84-787c-4e88-b7f1-fea3975ec850-kube-api-access-jtkft\") pod \"odh-model-controller-769875db56-fnf5k\" (UID: \"799aca84-787c-4e88-b7f1-fea3975ec850\") " pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:23.071501 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:23.071406 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:23.100767 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:23.100740 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pxv8z"] Nov 02 19:34:23.103410 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:23.103381 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3d028b_c7ed_4757_9161_6b55dc780ff8.slice/crio-2fef5f9acce0e9a8603decd0148a91218c8225be679b0c61c3d78a42059fa92d WatchSource:0}: Error finding container 2fef5f9acce0e9a8603decd0148a91218c8225be679b0c61c3d78a42059fa92d: Status 404 returned error can't find the container with id 2fef5f9acce0e9a8603decd0148a91218c8225be679b0c61c3d78a42059fa92d Nov 02 19:34:23.210819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:23.210797 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz"] Nov 02 19:34:23.213345 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:23.213291 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c8be9b5_e744_400a_9c31_18cd6c3b4975.slice/crio-ca1af668352412b0859bb0927841791e618f6186f5ee57f0ee4d9daf1f0e2e34 WatchSource:0}: Error finding container ca1af668352412b0859bb0927841791e618f6186f5ee57f0ee4d9daf1f0e2e34: Status 404 returned error can't find the container with id ca1af668352412b0859bb0927841791e618f6186f5ee57f0ee4d9daf1f0e2e34 Nov 02 19:34:23.264510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:23.264475 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" event={"ID":"fa3d028b-c7ed-4757-9161-6b55dc780ff8","Type":"ContainerStarted","Data":"2fef5f9acce0e9a8603decd0148a91218c8225be679b0c61c3d78a42059fa92d"} Nov 02 19:34:23.265782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:23.265754 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" event={"ID":"4c8be9b5-e744-400a-9c31-18cd6c3b4975","Type":"ContainerStarted","Data":"ca1af668352412b0859bb0927841791e618f6186f5ee57f0ee4d9daf1f0e2e34"} Nov 02 19:34:23.435716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:23.435690 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fnf5k"] Nov 02 19:34:23.437767 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:23.437743 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799aca84_787c_4e88_b7f1_fea3975ec850.slice/crio-0876a14baff54daab66f0c403a58b510b69836cf19245aa43416fab5d2e847e0 WatchSource:0}: Error finding container 0876a14baff54daab66f0c403a58b510b69836cf19245aa43416fab5d2e847e0: Status 404 returned error can't find the container with id 0876a14baff54daab66f0c403a58b510b69836cf19245aa43416fab5d2e847e0 Nov 02 19:34:24.274196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:24.274152 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fnf5k" event={"ID":"799aca84-787c-4e88-b7f1-fea3975ec850","Type":"ContainerStarted","Data":"0876a14baff54daab66f0c403a58b510b69836cf19245aa43416fab5d2e847e0"} Nov 02 19:34:27.299407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:27.299367 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" event={"ID":"fa3d028b-c7ed-4757-9161-6b55dc780ff8","Type":"ContainerStarted","Data":"d9fea81293660a6f827326889e78c8fe5d23d092608fda449c80183ce8c7fb8f"} Nov 02 19:34:27.299907 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:27.299484 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:27.301053 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:27.301028 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" event={"ID":"4c8be9b5-e744-400a-9c31-18cd6c3b4975","Type":"ContainerStarted","Data":"8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214"} Nov 02 19:34:27.301180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:27.301154 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:27.407550 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:27.407498 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" podStartSLOduration=1.8411961620000001 podStartE2EDuration="5.407482849s" totalImagesPullingTime="3.566286687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:27.405246599 +0000 UTC m=+373.353902438" watchObservedRunningTime="2026-11-02 19:34:27.407482849 +0000 UTC m=+373.356138712" Nov 02 19:34:27.511690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:27.511632 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" podStartSLOduration=1.782314012 podStartE2EDuration="5.511620087s" totalImagesPullingTime="3.729306075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:27.510448069 +0000 UTC m=+373.459103920" watchObservedRunningTime="2026-11-02 19:34:27.511620087 +0000 UTC m=+373.460275904" Nov 02 19:34:28.308835 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:28.308780 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fnf5k" event={"ID":"799aca84-787c-4e88-b7f1-fea3975ec850","Type":"ContainerStarted","Data":"38a8491e599c21c6ca4bc4b7d9de677add7ddc45cbf8dcb6d7eac4b797f2a5e8"} Nov 02 19:34:28.309254 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:28.309151 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:28.417038 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:28.416982 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-fnf5k" podStartSLOduration=2.173429982 podStartE2EDuration="6.416968545s" totalImagesPullingTime="4.243538563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:28.414801227 +0000 UTC m=+374.363457065" watchObservedRunningTime="2026-11-02 19:34:28.416968545 +0000 UTC m=+374.365624412" Nov 02 19:34:33.272006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:33.271963 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rvzdb" Nov 02 19:34:39.316734 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:39.316701 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-fnf5k" Nov 02 19:34:44.770227 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:44.770186 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:34:51.172469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.172432 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vxwh5"] Nov 02 19:34:51.197701 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.197674 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" Nov 02 19:34:51.230626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.230596 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vxwh5"] Nov 02 19:34:51.306551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.306511 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzjm\" (UniqueName: \"kubernetes.io/projected/6bfd741e-669d-42ea-a17b-af60d8454f7b-kube-api-access-tzzjm\") pod \"authorino-6f6944bcdd-vxwh5\" (UID: \"6bfd741e-669d-42ea-a17b-af60d8454f7b\") " pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" Nov 02 19:34:51.407500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.407454 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzjm\" (UniqueName: \"kubernetes.io/projected/6bfd741e-669d-42ea-a17b-af60d8454f7b-kube-api-access-tzzjm\") pod \"authorino-6f6944bcdd-vxwh5\" (UID: \"6bfd741e-669d-42ea-a17b-af60d8454f7b\") " pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" Nov 02 19:34:51.461958 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.461887 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzjm\" (UniqueName: \"kubernetes.io/projected/6bfd741e-669d-42ea-a17b-af60d8454f7b-kube-api-access-tzzjm\") pod \"authorino-6f6944bcdd-vxwh5\" (UID: \"6bfd741e-669d-42ea-a17b-af60d8454f7b\") " pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" Nov 02 19:34:51.510173 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.510134 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" Nov 02 19:34:51.519973 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.519946 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vxwh5"] Nov 02 19:34:51.541101 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.541067 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fb59c5c66-n8hgj"] Nov 02 19:34:51.546684 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.546656 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" Nov 02 19:34:51.597735 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.597703 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fb59c5c66-n8hgj"] Nov 02 19:34:51.609039 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.608999 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvhm\" (UniqueName: \"kubernetes.io/projected/239dc5ef-4545-4a09-b490-1a6f47701c48-kube-api-access-9fvhm\") pod \"authorino-6fb59c5c66-n8hgj\" (UID: \"239dc5ef-4545-4a09-b490-1a6f47701c48\") " pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" Nov 02 19:34:51.710391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.710357 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvhm\" (UniqueName: \"kubernetes.io/projected/239dc5ef-4545-4a09-b490-1a6f47701c48-kube-api-access-9fvhm\") pod \"authorino-6fb59c5c66-n8hgj\" (UID: \"239dc5ef-4545-4a09-b490-1a6f47701c48\") " pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" Nov 02 19:34:51.801662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.801626 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvhm\" (UniqueName: \"kubernetes.io/projected/239dc5ef-4545-4a09-b490-1a6f47701c48-kube-api-access-9fvhm\") pod \"authorino-6fb59c5c66-n8hgj\" (UID: \"239dc5ef-4545-4a09-b490-1a6f47701c48\") " pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" Nov 02 19:34:51.863322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.863280 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" Nov 02 19:34:51.873548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.873518 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fb59c5c66-n8hgj"] Nov 02 19:34:51.883574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.883536 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vxwh5"] Nov 02 19:34:51.885961 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:51.885924 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bfd741e_669d_42ea_a17b_af60d8454f7b.slice/crio-20d976246af2244fc1b1831f72b10891e1951982ccda152071361b57e717ae62 WatchSource:0}: Error finding container 20d976246af2244fc1b1831f72b10891e1951982ccda152071361b57e717ae62: Status 404 returned error can't find the container with id 20d976246af2244fc1b1831f72b10891e1951982ccda152071361b57e717ae62 Nov 02 19:34:51.899650 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.899622 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5554bdf764-ddvmt"] Nov 02 19:34:51.904553 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.904534 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:51.954040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.953981 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 19:34:51.956567 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:51.956540 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5554bdf764-ddvmt"] Nov 02 19:34:52.013374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.013344 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn9qd\" (UniqueName: \"kubernetes.io/projected/ced854fe-adae-4739-ba0e-55295de975d5-kube-api-access-pn9qd\") pod \"authorino-5554bdf764-ddvmt\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:52.013541 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.013392 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced854fe-adae-4739-ba0e-55295de975d5-tls-cert\") pod \"authorino-5554bdf764-ddvmt\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:52.114667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.114573 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced854fe-adae-4739-ba0e-55295de975d5-tls-cert\") pod \"authorino-5554bdf764-ddvmt\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:52.116824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.114669 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn9qd\" (UniqueName: \"kubernetes.io/projected/ced854fe-adae-4739-ba0e-55295de975d5-kube-api-access-pn9qd\") pod \"authorino-5554bdf764-ddvmt\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:52.117148 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.117123 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced854fe-adae-4739-ba0e-55295de975d5-tls-cert\") pod \"authorino-5554bdf764-ddvmt\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:52.167945 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.167919 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn9qd\" (UniqueName: \"kubernetes.io/projected/ced854fe-adae-4739-ba0e-55295de975d5-kube-api-access-pn9qd\") pod \"authorino-5554bdf764-ddvmt\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:52.219418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.219393 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:34:52.235569 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.235547 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fb59c5c66-n8hgj"] Nov 02 19:34:52.237244 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:34:52.237221 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod239dc5ef_4545_4a09_b490_1a6f47701c48.slice/crio-3572b092c2fa73e93d94b50e660eedf2a9bcdf08c088843e75db61dfbbac390e WatchSource:0}: Error finding container 3572b092c2fa73e93d94b50e660eedf2a9bcdf08c088843e75db61dfbbac390e: Status 404 returned error can't find the container with id 3572b092c2fa73e93d94b50e660eedf2a9bcdf08c088843e75db61dfbbac390e Nov 02 19:34:52.439720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.439613 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" event={"ID":"6bfd741e-669d-42ea-a17b-af60d8454f7b","Type":"ContainerStarted","Data":"20d976246af2244fc1b1831f72b10891e1951982ccda152071361b57e717ae62"} Nov 02 19:34:52.441220 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.441190 4966 generic.go:356] "Generic (PLEG): container finished" podID="c135774f-3f52-4d4c-81cb-aa2c86ea37d5" containerID="6ad127fcf2866f74406f343d7db81e8b32778e84ffa49e0c9d35021b0e8249d7" exitCode=0 Nov 02 19:34:52.441380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.441270 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" event={"ID":"c135774f-3f52-4d4c-81cb-aa2c86ea37d5","Type":"ContainerDied","Data":"6ad127fcf2866f74406f343d7db81e8b32778e84ffa49e0c9d35021b0e8249d7"} Nov 02 19:34:52.441682 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.441661 4966 scope.go:122] "RemoveContainer" containerID="6ad127fcf2866f74406f343d7db81e8b32778e84ffa49e0c9d35021b0e8249d7" Nov 02 19:34:52.442500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.442473 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" event={"ID":"239dc5ef-4545-4a09-b490-1a6f47701c48","Type":"ContainerStarted","Data":"3572b092c2fa73e93d94b50e660eedf2a9bcdf08c088843e75db61dfbbac390e"} Nov 02 19:34:52.442865 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.442835 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:34:52.442946 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.442920 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:34:52.443005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.442968 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:34:52.583879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:52.583841 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5554bdf764-ddvmt"] Nov 02 19:34:53.469361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:53.469327 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" event={"ID":"c135774f-3f52-4d4c-81cb-aa2c86ea37d5","Type":"ContainerStarted","Data":"b76da8f242871abdfd46a3a9ae902b4b933ee2cfd3c01105c0d0f1c1830cf0fb"} Nov 02 19:34:53.470683 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:53.470655 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5554bdf764-ddvmt" event={"ID":"ced854fe-adae-4739-ba0e-55295de975d5","Type":"ContainerStarted","Data":"fcf4a053f0a00d1231d8e127d71b01ca239fd2ac49fb7344f221dbc3fb68029f"} Nov 02 19:34:54.166258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.166217 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:34:54.167417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.167385 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-54rpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 19:34:54.167561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.167445 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" podUID="c135774f-3f52-4d4c-81cb-aa2c86ea37d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 19:34:54.359285 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.359194 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:34:54.487898 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.487855 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" event={"ID":"6bfd741e-669d-42ea-a17b-af60d8454f7b","Type":"ContainerStarted","Data":"1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7"} Nov 02 19:34:54.488573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.488469 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" podUID="6bfd741e-669d-42ea-a17b-af60d8454f7b" containerName="authorino" containerID="cri-o://1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7" gracePeriod=30 Nov 02 19:34:54.491913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.491886 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" event={"ID":"239dc5ef-4545-4a09-b490-1a6f47701c48","Type":"ContainerStarted","Data":"aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e"} Nov 02 19:34:54.492309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.492249 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" podUID="239dc5ef-4545-4a09-b490-1a6f47701c48" containerName="authorino" containerID="cri-o://aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e" gracePeriod=30 Nov 02 19:34:54.500569 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.500540 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5554bdf764-ddvmt" event={"ID":"ced854fe-adae-4739-ba0e-55295de975d5","Type":"ContainerStarted","Data":"476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f"} Nov 02 19:34:54.594438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.594373 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" podStartSLOduration=1.676173816 podStartE2EDuration="3.594355119s" totalImagesPullingTime="1.918181303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:54.592858823 +0000 UTC m=+400.541514663" watchObservedRunningTime="2026-11-02 19:34:54.594355119 +0000 UTC m=+400.543010960" Nov 02 19:34:54.720464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.720400 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5554bdf764-ddvmt" podStartSLOduration=2.366458632 podStartE2EDuration="3.720381164s" totalImagesPullingTime="1.353922532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:54.717246486 +0000 UTC m=+400.665902325" watchObservedRunningTime="2026-11-02 19:34:54.720381164 +0000 UTC m=+400.669037002" Nov 02 19:34:54.758984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.758945 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j62f7"] Nov 02 19:34:54.759526 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.759494 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" podUID="67e8cde7-0a82-4c99-af88-772a9bf0006c" containerName="authorino" containerID="cri-o://16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c" gracePeriod=30 Nov 02 19:34:54.831199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.826948 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" podStartSLOduration=2.33390308 podStartE2EDuration="3.826927342s" totalImagesPullingTime="1.493024262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:34:54.824547984 +0000 UTC m=+400.773203825" watchObservedRunningTime="2026-11-02 19:34:54.826927342 +0000 UTC m=+400.775583180" Nov 02 19:34:54.909798 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.909768 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" Nov 02 19:34:54.913488 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.913464 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" Nov 02 19:34:54.971751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.971197 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfd741e-669d-42ea-a17b-af60d8454f7b-kube-api-access-tzzjm\" (UniqueName: \"kubernetes.io/projected/6bfd741e-669d-42ea-a17b-af60d8454f7b-kube-api-access-tzzjm\") pod \"6bfd741e-669d-42ea-a17b-af60d8454f7b\" (UID: \"6bfd741e-669d-42ea-a17b-af60d8454f7b\") " Nov 02 19:34:54.971751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.971251 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239dc5ef-4545-4a09-b490-1a6f47701c48-kube-api-access-9fvhm\" (UniqueName: \"kubernetes.io/projected/239dc5ef-4545-4a09-b490-1a6f47701c48-kube-api-access-9fvhm\") pod \"239dc5ef-4545-4a09-b490-1a6f47701c48\" (UID: \"239dc5ef-4545-4a09-b490-1a6f47701c48\") " Nov 02 19:34:54.973623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.973593 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfd741e-669d-42ea-a17b-af60d8454f7b-kube-api-access-tzzjm" pod "6bfd741e-669d-42ea-a17b-af60d8454f7b" (UID: "6bfd741e-669d-42ea-a17b-af60d8454f7b"). InnerVolumeSpecName "kube-api-access-tzzjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:34:54.973756 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:54.973740 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239dc5ef-4545-4a09-b490-1a6f47701c48-kube-api-access-9fvhm" pod "239dc5ef-4545-4a09-b490-1a6f47701c48" (UID: "239dc5ef-4545-4a09-b490-1a6f47701c48"). InnerVolumeSpecName "kube-api-access-9fvhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:34:55.078330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.076612 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzzjm\" (UniqueName: \"kubernetes.io/projected/6bfd741e-669d-42ea-a17b-af60d8454f7b-kube-api-access-tzzjm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:55.078330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.076651 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fvhm\" (UniqueName: \"kubernetes.io/projected/239dc5ef-4545-4a09-b490-1a6f47701c48-kube-api-access-9fvhm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:55.111327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.110436 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" Nov 02 19:34:55.171856 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.169688 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-54rpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 19:34:55.171856 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.169754 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" podUID="c135774f-3f52-4d4c-81cb-aa2c86ea37d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 19:34:55.177190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.177105 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e8cde7-0a82-4c99-af88-772a9bf0006c-kube-api-access-rqsgp\" (UniqueName: \"kubernetes.io/projected/67e8cde7-0a82-4c99-af88-772a9bf0006c-kube-api-access-rqsgp\") pod \"67e8cde7-0a82-4c99-af88-772a9bf0006c\" (UID: \"67e8cde7-0a82-4c99-af88-772a9bf0006c\") " Nov 02 19:34:55.179431 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.179409 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e8cde7-0a82-4c99-af88-772a9bf0006c-kube-api-access-rqsgp" pod "67e8cde7-0a82-4c99-af88-772a9bf0006c" (UID: "67e8cde7-0a82-4c99-af88-772a9bf0006c"). InnerVolumeSpecName "kube-api-access-rqsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:34:55.279111 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.278681 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqsgp\" (UniqueName: \"kubernetes.io/projected/67e8cde7-0a82-4c99-af88-772a9bf0006c-kube-api-access-rqsgp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:34:55.506953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.506912 4966 generic.go:356] "Generic (PLEG): container finished" podID="6bfd741e-669d-42ea-a17b-af60d8454f7b" containerID="1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7" exitCode=0 Nov 02 19:34:55.507423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.506959 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" Nov 02 19:34:55.507423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.506999 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" event={"ID":"6bfd741e-669d-42ea-a17b-af60d8454f7b","Type":"ContainerDied","Data":"1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7"} Nov 02 19:34:55.507423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.507045 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-vxwh5" event={"ID":"6bfd741e-669d-42ea-a17b-af60d8454f7b","Type":"ContainerDied","Data":"20d976246af2244fc1b1831f72b10891e1951982ccda152071361b57e717ae62"} Nov 02 19:34:55.507423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.507062 4966 scope.go:122] "RemoveContainer" containerID="1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7" Nov 02 19:34:55.508522 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.508499 4966 generic.go:356] "Generic (PLEG): container finished" podID="67e8cde7-0a82-4c99-af88-772a9bf0006c" containerID="16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c" exitCode=0 Nov 02 19:34:55.508633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.508533 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" event={"ID":"67e8cde7-0a82-4c99-af88-772a9bf0006c","Type":"ContainerDied","Data":"16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c"} Nov 02 19:34:55.508633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.508558 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" Nov 02 19:34:55.508633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.508583 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-j62f7" event={"ID":"67e8cde7-0a82-4c99-af88-772a9bf0006c","Type":"ContainerDied","Data":"f785d1d70d42d6a049350c46bb6ffbea2214824e3fdc0db3bb729f664625c16c"} Nov 02 19:34:55.509919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.509896 4966 generic.go:356] "Generic (PLEG): container finished" podID="239dc5ef-4545-4a09-b490-1a6f47701c48" containerID="aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e" exitCode=0 Nov 02 19:34:55.510015 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.509929 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" Nov 02 19:34:55.510015 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.509981 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" event={"ID":"239dc5ef-4545-4a09-b490-1a6f47701c48","Type":"ContainerDied","Data":"aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e"} Nov 02 19:34:55.510099 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.510015 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb59c5c66-n8hgj" event={"ID":"239dc5ef-4545-4a09-b490-1a6f47701c48","Type":"ContainerDied","Data":"3572b092c2fa73e93d94b50e660eedf2a9bcdf08c088843e75db61dfbbac390e"} Nov 02 19:34:55.518311 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.518281 4966 scope.go:122] "RemoveContainer" containerID="1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7" Nov 02 19:34:55.518633 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:55.518609 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7\": container with ID starting with 1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7 not found: ID does not exist" containerID="1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7" Nov 02 19:34:55.518766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.518646 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7"} err="failed to get container status \"1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7\": rpc error: code = NotFound desc = could not find container \"1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7\": container with ID starting with 1e3db46b1f99bcb587284afa710e149c4fe96e7aff85e6d7c6c0fa2a8a151da7 not found: ID does not exist" Nov 02 19:34:55.518766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.518669 4966 scope.go:122] "RemoveContainer" containerID="16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c" Nov 02 19:34:55.529868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.529835 4966 scope.go:122] "RemoveContainer" containerID="16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c" Nov 02 19:34:55.530131 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:55.530111 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c\": container with ID starting with 16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c not found: ID does not exist" containerID="16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c" Nov 02 19:34:55.530194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.530137 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c"} err="failed to get container status \"16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c\": rpc error: code = NotFound desc = could not find container \"16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c\": container with ID starting with 16d15670d6fc1e8a13a86620deeb7dece9372f0e6883e9905a46ab946345ca2c not found: ID does not exist" Nov 02 19:34:55.530194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.530152 4966 scope.go:122] "RemoveContainer" containerID="aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e" Nov 02 19:34:55.560471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.554658 4966 scope.go:122] "RemoveContainer" containerID="aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e" Nov 02 19:34:55.562445 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:34:55.562409 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e\": container with ID starting with aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e not found: ID does not exist" containerID="aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e" Nov 02 19:34:55.562562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.562452 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e"} err="failed to get container status \"aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e\": rpc error: code = NotFound desc = could not find container \"aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e\": container with ID starting with aae66ca47b8077245bc0a29fcd4d47050823fcb4c116c44b897a85a4584a577e not found: ID does not exist" Nov 02 19:34:55.663629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.663587 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fb59c5c66-n8hgj"] Nov 02 19:34:55.688862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.688586 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fb59c5c66-n8hgj"] Nov 02 19:34:55.765768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.765678 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vxwh5"] Nov 02 19:34:55.768930 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.768903 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vxwh5"] Nov 02 19:34:55.869100 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.869060 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j62f7"] Nov 02 19:34:55.871150 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:55.871124 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-j62f7"] Nov 02 19:34:56.167154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:56.167036 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-54rpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 19:34:56.167154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:56.167129 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" podUID="c135774f-3f52-4d4c-81cb-aa2c86ea37d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 19:34:56.183213 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:56.183180 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239dc5ef-4545-4a09-b490-1a6f47701c48" path="/var/lib/kubelet/pods/239dc5ef-4545-4a09-b490-1a6f47701c48/volumes" Nov 02 19:34:56.183584 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:56.183568 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e8cde7-0a82-4c99-af88-772a9bf0006c" path="/var/lib/kubelet/pods/67e8cde7-0a82-4c99-af88-772a9bf0006c/volumes" Nov 02 19:34:56.183868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:56.183856 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfd741e-669d-42ea-a17b-af60d8454f7b" path="/var/lib/kubelet/pods/6bfd741e-669d-42ea-a17b-af60d8454f7b/volumes" Nov 02 19:34:57.166085 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:57.166042 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-54rpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 19:34:57.166525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:57.166123 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" podUID="c135774f-3f52-4d4c-81cb-aa2c86ea37d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 19:34:58.166212 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:58.166166 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-54rpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 19:34:58.166637 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:58.166253 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" podUID="c135774f-3f52-4d4c-81cb-aa2c86ea37d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 19:34:58.311125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:58.311092 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:34:58.313953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:58.313932 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pxv8z" Nov 02 19:34:59.170174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:59.170146 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:34:59.171118 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:34:59.171099 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-54rpb" Nov 02 19:35:14.751255 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:35:14.751215 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:35:36.001749 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.001712 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm"] Nov 02 19:35:36.002181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002137 4966 cpu_manager.go:414] "containerMap: removing container" podUID="67e8cde7-0a82-4c99-af88-772a9bf0006c" containerName="authorino" Nov 02 19:35:36.002181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002148 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e8cde7-0a82-4c99-af88-772a9bf0006c" containerName="authorino" Nov 02 19:35:36.002181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002167 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6bfd741e-669d-42ea-a17b-af60d8454f7b" containerName="authorino" Nov 02 19:35:36.002181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002173 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfd741e-669d-42ea-a17b-af60d8454f7b" containerName="authorino" Nov 02 19:35:36.002327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002199 4966 cpu_manager.go:414] "containerMap: removing container" podUID="239dc5ef-4545-4a09-b490-1a6f47701c48" containerName="authorino" Nov 02 19:35:36.002327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002204 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239dc5ef-4545-4a09-b490-1a6f47701c48" containerName="authorino" Nov 02 19:35:36.002327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002256 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e8cde7-0a82-4c99-af88-772a9bf0006c" containerName="authorino" Nov 02 19:35:36.002327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002264 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfd741e-669d-42ea-a17b-af60d8454f7b" containerName="authorino" Nov 02 19:35:36.002327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.002275 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="239dc5ef-4545-4a09-b490-1a6f47701c48" containerName="authorino" Nov 02 19:35:36.005482 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.005466 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.051824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.051789 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8b1d15-8458-4c37-8da8-b84182728c1c-cert\") pod \"llmisvc-controller-manager-6466c554d8-wcnxm\" (UID: \"cb8b1d15-8458-4c37-8da8-b84182728c1c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.051982 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.051924 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqjn\" (UniqueName: \"kubernetes.io/projected/cb8b1d15-8458-4c37-8da8-b84182728c1c-kube-api-access-2gqjn\") pod \"llmisvc-controller-manager-6466c554d8-wcnxm\" (UID: \"cb8b1d15-8458-4c37-8da8-b84182728c1c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.058802 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.058775 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm"] Nov 02 19:35:36.152833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.152793 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqjn\" (UniqueName: \"kubernetes.io/projected/cb8b1d15-8458-4c37-8da8-b84182728c1c-kube-api-access-2gqjn\") pod \"llmisvc-controller-manager-6466c554d8-wcnxm\" (UID: \"cb8b1d15-8458-4c37-8da8-b84182728c1c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.153011 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.152891 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8b1d15-8458-4c37-8da8-b84182728c1c-cert\") pod \"llmisvc-controller-manager-6466c554d8-wcnxm\" (UID: \"cb8b1d15-8458-4c37-8da8-b84182728c1c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.155550 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.155523 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8b1d15-8458-4c37-8da8-b84182728c1c-cert\") pod \"llmisvc-controller-manager-6466c554d8-wcnxm\" (UID: \"cb8b1d15-8458-4c37-8da8-b84182728c1c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.206005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.205975 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqjn\" (UniqueName: \"kubernetes.io/projected/cb8b1d15-8458-4c37-8da8-b84182728c1c-kube-api-access-2gqjn\") pod \"llmisvc-controller-manager-6466c554d8-wcnxm\" (UID: \"cb8b1d15-8458-4c37-8da8-b84182728c1c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.318093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.318009 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:36.680045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.680016 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm"] Nov 02 19:35:36.682173 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:35:36.682142 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8b1d15_8458_4c37_8da8_b84182728c1c.slice/crio-b708e6acffc82754651e709e07da4858721053f854ebb6a30dd6f059629581d0 WatchSource:0}: Error finding container b708e6acffc82754651e709e07da4858721053f854ebb6a30dd6f059629581d0: Status 404 returned error can't find the container with id b708e6acffc82754651e709e07da4858721053f854ebb6a30dd6f059629581d0 Nov 02 19:35:36.741108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:36.741075 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" event={"ID":"cb8b1d15-8458-4c37-8da8-b84182728c1c","Type":"ContainerStarted","Data":"b708e6acffc82754651e709e07da4858721053f854ebb6a30dd6f059629581d0"} Nov 02 19:35:37.748182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:37.748090 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" event={"ID":"cb8b1d15-8458-4c37-8da8-b84182728c1c","Type":"ContainerStarted","Data":"809dd637557b2fd7788edf2ae92c0c9c9a63598054aceb8a2e1032195e16b215"} Nov 02 19:35:37.748647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:37.748244 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:35:37.855725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:35:37.855672 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" podStartSLOduration=2.069029291 podStartE2EDuration="2.855657615s" totalImagesPullingTime="786.628324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:35:37.853403523 +0000 UTC m=+443.802059363" watchObservedRunningTime="2026-11-02 19:35:37.855657615 +0000 UTC m=+443.804313432" Nov 02 19:35:44.751186 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:35:44.751148 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:36:08.756147 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:08.756115 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wcnxm" Nov 02 19:36:08.910780 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:08.910745 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz"] Nov 02 19:36:08.911048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:08.910990 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" podUID="4c8be9b5-e744-400a-9c31-18cd6c3b4975" containerName="manager" containerID="cri-o://8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214" gracePeriod=30 Nov 02 19:36:09.162987 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.162962 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:36:09.249737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.249706 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c8be9b5-e744-400a-9c31-18cd6c3b4975-cert\" (UniqueName: \"kubernetes.io/secret/4c8be9b5-e744-400a-9c31-18cd6c3b4975-cert\") pod \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " Nov 02 19:36:09.249915 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.249764 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8be9b5-e744-400a-9c31-18cd6c3b4975-kube-api-access-ks9j4\" (UniqueName: \"kubernetes.io/projected/4c8be9b5-e744-400a-9c31-18cd6c3b4975-kube-api-access-ks9j4\") pod \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\" (UID: \"4c8be9b5-e744-400a-9c31-18cd6c3b4975\") " Nov 02 19:36:09.251902 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.251873 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c8be9b5-e744-400a-9c31-18cd6c3b4975-cert" pod "4c8be9b5-e744-400a-9c31-18cd6c3b4975" (UID: "4c8be9b5-e744-400a-9c31-18cd6c3b4975"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:36:09.252044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.251970 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8be9b5-e744-400a-9c31-18cd6c3b4975-kube-api-access-ks9j4" pod "4c8be9b5-e744-400a-9c31-18cd6c3b4975" (UID: "4c8be9b5-e744-400a-9c31-18cd6c3b4975"). InnerVolumeSpecName "kube-api-access-ks9j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:36:09.350476 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.350384 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks9j4\" (UniqueName: \"kubernetes.io/projected/4c8be9b5-e744-400a-9c31-18cd6c3b4975-kube-api-access-ks9j4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:36:09.350476 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.350416 4966 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8be9b5-e744-400a-9c31-18cd6c3b4975-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:36:09.921517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.921476 4966 generic.go:356] "Generic (PLEG): container finished" podID="4c8be9b5-e744-400a-9c31-18cd6c3b4975" containerID="8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214" exitCode=0 Nov 02 19:36:09.922044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.921524 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" Nov 02 19:36:09.922044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.921553 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" event={"ID":"4c8be9b5-e744-400a-9c31-18cd6c3b4975","Type":"ContainerDied","Data":"8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214"} Nov 02 19:36:09.922044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.921607 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz" event={"ID":"4c8be9b5-e744-400a-9c31-18cd6c3b4975","Type":"ContainerDied","Data":"ca1af668352412b0859bb0927841791e618f6186f5ee57f0ee4d9daf1f0e2e34"} Nov 02 19:36:09.922044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.921630 4966 scope.go:122] "RemoveContainer" containerID="8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214" Nov 02 19:36:09.931158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.931140 4966 scope.go:122] "RemoveContainer" containerID="8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214" Nov 02 19:36:09.931437 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:36:09.931418 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214\": container with ID starting with 8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214 not found: ID does not exist" containerID="8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214" Nov 02 19:36:09.931519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:09.931445 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214"} err="failed to get container status \"8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214\": rpc error: code = NotFound desc = could not find container \"8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214\": container with ID starting with 8e619adab7138cae8948b1e5324162592d62c3db93cbaf5fe0e9edba30523214 not found: ID does not exist" Nov 02 19:36:10.083601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:10.083567 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz"] Nov 02 19:36:10.130155 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:10.130122 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qdfsz"] Nov 02 19:36:10.182225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:36:10.182143 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8be9b5-e744-400a-9c31-18cd6c3b4975" path="/var/lib/kubelet/pods/4c8be9b5-e744-400a-9c31-18cd6c3b4975/volumes" Nov 02 19:36:14.750948 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:36:14.750909 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:36:44.750678 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:36:44.750642 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:37:14.751369 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:37:14.751329 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:37:24.734782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.734748 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5846d6f4ff-m9hwk"] Nov 02 19:37:24.735244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.735132 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4c8be9b5-e744-400a-9c31-18cd6c3b4975" containerName="manager" Nov 02 19:37:24.735244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.735142 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8be9b5-e744-400a-9c31-18cd6c3b4975" containerName="manager" Nov 02 19:37:24.735244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.735232 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8be9b5-e744-400a-9c31-18cd6c3b4975" containerName="manager" Nov 02 19:37:24.738786 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.738764 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:24.791445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.791418 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5846d6f4ff-m9hwk"] Nov 02 19:37:24.817327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.817263 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b60e0ac4-f6f1-47ed-b31f-fcf481449ba7-tls-cert\") pod \"authorino-5846d6f4ff-m9hwk\" (UID: \"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7\") " pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:24.817504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.817424 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sh7s\" (UniqueName: \"kubernetes.io/projected/b60e0ac4-f6f1-47ed-b31f-fcf481449ba7-kube-api-access-6sh7s\") pod \"authorino-5846d6f4ff-m9hwk\" (UID: \"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7\") " pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:24.918511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.918477 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b60e0ac4-f6f1-47ed-b31f-fcf481449ba7-tls-cert\") pod \"authorino-5846d6f4ff-m9hwk\" (UID: \"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7\") " pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:24.918676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.918525 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sh7s\" (UniqueName: \"kubernetes.io/projected/b60e0ac4-f6f1-47ed-b31f-fcf481449ba7-kube-api-access-6sh7s\") pod \"authorino-5846d6f4ff-m9hwk\" (UID: \"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7\") " pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:24.920926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.920906 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b60e0ac4-f6f1-47ed-b31f-fcf481449ba7-tls-cert\") pod \"authorino-5846d6f4ff-m9hwk\" (UID: \"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7\") " pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:24.972026 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:24.971996 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sh7s\" (UniqueName: \"kubernetes.io/projected/b60e0ac4-f6f1-47ed-b31f-fcf481449ba7-kube-api-access-6sh7s\") pod \"authorino-5846d6f4ff-m9hwk\" (UID: \"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7\") " pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:25.051107 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:25.051035 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" Nov 02 19:37:25.412999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:25.412974 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5846d6f4ff-m9hwk"] Nov 02 19:37:25.415133 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:37:25.415101 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb60e0ac4_f6f1_47ed_b31f_fcf481449ba7.slice/crio-9376bb0db43de6912b32fb4ef81d3a17fbdadc93d509347f2761a02d8d3b1551 WatchSource:0}: Error finding container 9376bb0db43de6912b32fb4ef81d3a17fbdadc93d509347f2761a02d8d3b1551: Status 404 returned error can't find the container with id 9376bb0db43de6912b32fb4ef81d3a17fbdadc93d509347f2761a02d8d3b1551 Nov 02 19:37:26.358731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:26.358698 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" event={"ID":"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7","Type":"ContainerStarted","Data":"9376bb0db43de6912b32fb4ef81d3a17fbdadc93d509347f2761a02d8d3b1551"} Nov 02 19:37:27.365614 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.365530 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" event={"ID":"b60e0ac4-f6f1-47ed-b31f-fcf481449ba7","Type":"ContainerStarted","Data":"9465b3f21227b15c6ad5d40b65f8f2f6b91e6d6e3fc60cef4dfacd0d9bde992c"} Nov 02 19:37:27.475348 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.475282 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5846d6f4ff-m9hwk" podStartSLOduration=1.803253422 podStartE2EDuration="3.475268852s" totalImagesPullingTime="1.67201543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:37:27.472692111 +0000 UTC m=+553.421347950" watchObservedRunningTime="2026-11-02 19:37:27.475268852 +0000 UTC m=+553.423924690" Nov 02 19:37:27.521443 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.521286 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5554bdf764-ddvmt"] Nov 02 19:37:27.521674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.521647 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5554bdf764-ddvmt" podUID="ced854fe-adae-4739-ba0e-55295de975d5" containerName="authorino" containerID="cri-o://476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f" gracePeriod=30 Nov 02 19:37:27.782309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.782272 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:37:27.946655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.946564 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced854fe-adae-4739-ba0e-55295de975d5-kube-api-access-pn9qd\" (UniqueName: \"kubernetes.io/projected/ced854fe-adae-4739-ba0e-55295de975d5-kube-api-access-pn9qd\") pod \"ced854fe-adae-4739-ba0e-55295de975d5\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " Nov 02 19:37:27.946655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.946625 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced854fe-adae-4739-ba0e-55295de975d5-tls-cert\" (UniqueName: \"kubernetes.io/secret/ced854fe-adae-4739-ba0e-55295de975d5-tls-cert\") pod \"ced854fe-adae-4739-ba0e-55295de975d5\" (UID: \"ced854fe-adae-4739-ba0e-55295de975d5\") " Nov 02 19:37:27.948867 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.948834 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced854fe-adae-4739-ba0e-55295de975d5-kube-api-access-pn9qd" pod "ced854fe-adae-4739-ba0e-55295de975d5" (UID: "ced854fe-adae-4739-ba0e-55295de975d5"). InnerVolumeSpecName "kube-api-access-pn9qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:37:27.958081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:27.958052 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced854fe-adae-4739-ba0e-55295de975d5-tls-cert" pod "ced854fe-adae-4739-ba0e-55295de975d5" (UID: "ced854fe-adae-4739-ba0e-55295de975d5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:37:28.047722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.047680 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn9qd\" (UniqueName: \"kubernetes.io/projected/ced854fe-adae-4739-ba0e-55295de975d5-kube-api-access-pn9qd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:37:28.047722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.047714 4966 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced854fe-adae-4739-ba0e-55295de975d5-tls-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:37:28.373145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.372526 4966 generic.go:356] "Generic (PLEG): container finished" podID="ced854fe-adae-4739-ba0e-55295de975d5" containerID="476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f" exitCode=0 Nov 02 19:37:28.373145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.372566 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5554bdf764-ddvmt" event={"ID":"ced854fe-adae-4739-ba0e-55295de975d5","Type":"ContainerDied","Data":"476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f"} Nov 02 19:37:28.373145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.372613 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5554bdf764-ddvmt" event={"ID":"ced854fe-adae-4739-ba0e-55295de975d5","Type":"ContainerDied","Data":"fcf4a053f0a00d1231d8e127d71b01ca239fd2ac49fb7344f221dbc3fb68029f"} Nov 02 19:37:28.373145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.372635 4966 scope.go:122] "RemoveContainer" containerID="476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f" Nov 02 19:37:28.373145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.372583 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5554bdf764-ddvmt" Nov 02 19:37:28.382550 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.382517 4966 scope.go:122] "RemoveContainer" containerID="476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f" Nov 02 19:37:28.382840 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:37:28.382818 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f\": container with ID starting with 476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f not found: ID does not exist" containerID="476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f" Nov 02 19:37:28.382912 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.382850 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f"} err="failed to get container status \"476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f\": rpc error: code = NotFound desc = could not find container \"476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f\": container with ID starting with 476f79f1444cca1b534a8c2431a709ebfc8dd69cacffe0d2362b916a803f8d6f not found: ID does not exist" Nov 02 19:37:28.524561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.524522 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5554bdf764-ddvmt"] Nov 02 19:37:28.570449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:28.570417 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5554bdf764-ddvmt"] Nov 02 19:37:30.181374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:37:30.181338 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced854fe-adae-4739-ba0e-55295de975d5" path="/var/lib/kubelet/pods/ced854fe-adae-4739-ba0e-55295de975d5/volumes" Nov 02 19:37:44.751323 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:37:44.751266 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:38:01.542117 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.542036 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc"] Nov 02 19:38:01.542643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.542532 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ced854fe-adae-4739-ba0e-55295de975d5" containerName="authorino" Nov 02 19:38:01.542643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.542546 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced854fe-adae-4739-ba0e-55295de975d5" containerName="authorino" Nov 02 19:38:01.542643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.542607 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced854fe-adae-4739-ba0e-55295de975d5" containerName="authorino" Nov 02 19:38:01.545330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.545313 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.643860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.643825 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc"] Nov 02 19:38:01.655318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655269 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655330 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjjnp\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-kube-api-access-kjjnp\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655415 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655454 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655484 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655517 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655570 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655599 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.655748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.655641 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757007 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.756970 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757007 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757009 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjjnp\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-kube-api-access-kjjnp\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757032 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757047 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757065 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757085 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757132 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757174 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757220 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757567 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757628 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757699 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757861 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757824 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.757950 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.757931 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.759546 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.759518 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.759687 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.759670 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.809994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.809931 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.810250 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.810226 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjjnp\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-kube-api-access-kjjnp\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-rzqnc\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:01.858428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:01.858399 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:02.276773 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.274822 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc"] Nov 02 19:38:02.278592 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:38:02.278562 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9671ce7_e5a4_4f0b_a6df_929cafc909a5.slice/crio-45a80be09a0dc29f24226636becfc7f7e3fee6f0e65b94b277c61f78a41c1e10 WatchSource:0}: Error finding container 45a80be09a0dc29f24226636becfc7f7e3fee6f0e65b94b277c61f78a41c1e10: Status 404 returned error can't find the container with id 45a80be09a0dc29f24226636becfc7f7e3fee6f0e65b94b277c61f78a41c1e10 Nov 02 19:38:02.280696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.280665 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:02.280754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.280728 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:02.280797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.280758 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:02.566826 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.566734 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" event={"ID":"e9671ce7-e5a4-4f0b-a6df-929cafc909a5","Type":"ContainerStarted","Data":"dd97edb115f57ddb7519669153e07d01752fe8c1b46b31803fa12747bf544328"} Nov 02 19:38:02.566826 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.566801 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" event={"ID":"e9671ce7-e5a4-4f0b-a6df-929cafc909a5","Type":"ContainerStarted","Data":"45a80be09a0dc29f24226636becfc7f7e3fee6f0e65b94b277c61f78a41c1e10"} Nov 02 19:38:02.676445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.676395 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podStartSLOduration=1.6763801040000001 podStartE2EDuration="1.676380104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:02.675116943 +0000 UTC m=+588.623772783" watchObservedRunningTime="2026-11-02 19:38:02.676380104 +0000 UTC m=+588.625035943" Nov 02 19:38:02.862572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:02.862458 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:03.864142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:03.864104 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 19:38:03.864601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:03.864187 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 19:38:04.862706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:04.862665 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 19:38:04.862922 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:04.862734 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 19:38:05.862760 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:05.862718 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 19:38:05.863158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:05.862785 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 19:38:06.859707 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:06.859667 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 19:38:06.860036 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:06.859996 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 19:38:07.862223 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:07.861867 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 19:38:07.862223 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:07.861925 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 19:38:08.859779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:08.859739 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 19:38:08.859958 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:08.859799 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 19:38:10.531535 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:10.531497 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:10.532064 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:10.531907 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:10.532788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:10.532768 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:38:14.238576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:14.238548 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:38:14.240352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:14.240330 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:38:14.251128 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:14.251107 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:38:14.252595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:14.252573 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:38:14.751555 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:38:14.751520 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:38:44.750955 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:38:44.750917 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:38:46.292808 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.292767 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx"] Nov 02 19:38:46.297779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.297752 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.363416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.363380 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx"] Nov 02 19:38:46.363763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.363731 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.363814 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.363786 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lp88\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-kube-api-access-4lp88\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.363909 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.363891 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.363956 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.363938 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.364038 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.364015 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.364166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.364149 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.364217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.364202 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.364271 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.364236 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.364364 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.364330 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.388905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.388870 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26cfbe-openshift-default-dockercfg-msctx\"" Nov 02 19:38:46.465522 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465486 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.465715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465537 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lp88\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-kube-api-access-4lp88\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.465715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465570 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.465715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465600 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.465715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465622 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.465715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465700 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465746 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465775 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.465801 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.466172 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.466493 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.466514 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.466612 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.466850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.466829 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.469060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.469039 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.469221 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.469201 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.518748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.518703 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.518927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.518857 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lp88\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-kube-api-access-4lp88\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:46.611173 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:46.611096 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:47.054575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.054540 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx"] Nov 02 19:38:47.056734 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:38:47.056709 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa43ea8b_4430_49d2_940a_dcd655dbc83e.slice/crio-4d1cdd4e831fdf638d71870cebb06f0c061ee89c99d32fe2f138ac38db85c5ba WatchSource:0}: Error finding container 4d1cdd4e831fdf638d71870cebb06f0c061ee89c99d32fe2f138ac38db85c5ba: Status 404 returned error can't find the container with id 4d1cdd4e831fdf638d71870cebb06f0c061ee89c99d32fe2f138ac38db85c5ba Nov 02 19:38:47.059103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.059059 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:47.059196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.059141 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:47.059196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.059185 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:47.287756 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.287721 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r"] Nov 02 19:38:47.292219 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.292188 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.349276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.348154 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r"] Nov 02 19:38:47.349740 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.349561 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-28623b-openshift-default-dockercfg-cxdg7\"" Nov 02 19:38:47.374933 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.374900 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.374989 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.375094 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0861dab-4be6-4d9e-b2bb-e7f214200037-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375262 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.375191 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375262 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.375249 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.375344 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.375373 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp6zv\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-kube-api-access-fp6zv\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375496 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.375427 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.375561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.375544 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476198 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476253 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp6zv\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-kube-api-access-fp6zv\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476289 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476354 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476393 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476482 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476514 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0861dab-4be6-4d9e-b2bb-e7f214200037-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476550 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476605 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.476849 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476753 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.477190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.476956 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.477455 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.477432 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.477934 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.477906 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0861dab-4be6-4d9e-b2bb-e7f214200037-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.478142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.478120 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.480371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.480176 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.480948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.480926 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.530654 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.530627 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp6zv\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-kube-api-access-fp6zv\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.530812 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.530633 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.607694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.607601 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:47.816575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.816540 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" event={"ID":"aa43ea8b-4430-49d2-940a-dcd655dbc83e","Type":"ContainerStarted","Data":"e93c9d49d20fd7570dd8464a758368a4f40284ae2ff48912c2740c8d840ed7b9"} Nov 02 19:38:47.816754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.816592 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" event={"ID":"aa43ea8b-4430-49d2-940a-dcd655dbc83e","Type":"ContainerStarted","Data":"4d1cdd4e831fdf638d71870cebb06f0c061ee89c99d32fe2f138ac38db85c5ba"} Nov 02 19:38:47.927768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:47.927659 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podStartSLOduration=1.927644376 podStartE2EDuration="1.927644376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:47.925358619 +0000 UTC m=+633.874014458" watchObservedRunningTime="2026-11-02 19:38:47.927644376 +0000 UTC m=+633.876300215" Nov 02 19:38:48.028124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.028096 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r"] Nov 02 19:38:48.030390 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:38:48.030360 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0861dab_4be6_4d9e_b2bb_e7f214200037.slice/crio-77bf809e28bf02918d4115ff17f7c7619fc6133eeee7e0c8e00671ddd463ad48 WatchSource:0}: Error finding container 77bf809e28bf02918d4115ff17f7c7619fc6133eeee7e0c8e00671ddd463ad48: Status 404 returned error can't find the container with id 77bf809e28bf02918d4115ff17f7c7619fc6133eeee7e0c8e00671ddd463ad48 Nov 02 19:38:48.033142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.033103 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:48.033276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.033195 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:48.033276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.033237 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:38:48.611348 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.611272 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:48.616689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.616659 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:48.824125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.824087 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" event={"ID":"c0861dab-4be6-4d9e-b2bb-e7f214200037","Type":"ContainerStarted","Data":"618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b"} Nov 02 19:38:48.824357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.824150 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" event={"ID":"c0861dab-4be6-4d9e-b2bb-e7f214200037","Type":"ContainerStarted","Data":"77bf809e28bf02918d4115ff17f7c7619fc6133eeee7e0c8e00671ddd463ad48"} Nov 02 19:38:48.824357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.824199 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:48.825274 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.825258 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:38:48.933318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:48.933175 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podStartSLOduration=1.9331598319999999 podStartE2EDuration="1.933159832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:38:48.930622344 +0000 UTC m=+634.879278183" watchObservedRunningTime="2026-11-02 19:38:48.933159832 +0000 UTC m=+634.881815762" Nov 02 19:38:49.608269 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:49.608230 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:49.613549 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:49.613525 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:49.830751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:49.830708 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:38:49.831782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:38:49.831758 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:39:02.334820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.334783 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc"] Nov 02 19:39:02.338759 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.338736 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:02.435540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.435503 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-43ceaa\"" Nov 02 19:39:02.435739 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.435569 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-43ceaa-dockercfg-zmpmn\"" Nov 02 19:39:02.438420 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.438393 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc"] Nov 02 19:39:02.440223 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.440198 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:02.440356 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.440256 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grffb\" (UniqueName: \"kubernetes.io/projected/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-kube-api-access-grffb\") pod \"payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:02.541395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.541356 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:02.541587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.541403 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grffb\" (UniqueName: \"kubernetes.io/projected/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-kube-api-access-grffb\") pod \"payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:02.542186 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.542140 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:02.594636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.594553 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grffb\" (UniqueName: \"kubernetes.io/projected/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-kube-api-access-grffb\") pod \"payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:02.653003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:02.652963 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:03.024659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:03.024628 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc"] Nov 02 19:39:03.025949 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:39:03.025914 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7499b11_1af4_4ca5_acef_68a5d6dcde9b.slice/crio-cdcb401c3d16fd35eeba0807702150c6e2ee514ba187ed58118bea5cbbd00a83 WatchSource:0}: Error finding container cdcb401c3d16fd35eeba0807702150c6e2ee514ba187ed58118bea5cbbd00a83: Status 404 returned error can't find the container with id cdcb401c3d16fd35eeba0807702150c6e2ee514ba187ed58118bea5cbbd00a83 Nov 02 19:39:03.028037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:03.028018 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:39:03.917247 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:03.917213 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" event={"ID":"d7499b11-1af4-4ca5-acef-68a5d6dcde9b","Type":"ContainerStarted","Data":"cdcb401c3d16fd35eeba0807702150c6e2ee514ba187ed58118bea5cbbd00a83"} Nov 02 19:39:05.020464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.020425 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct"] Nov 02 19:39:05.027381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.027349 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.091359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.078086 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct"] Nov 02 19:39:05.120882 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.120842 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-1d869e-dockercfg-7l89t\"" Nov 02 19:39:05.120882 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.120842 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-1d869e\"" Nov 02 19:39:05.183837 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.183801 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m55t\" (UniqueName: \"kubernetes.io/projected/bd2b3ee3-285e-4740-aee2-7003aa50baed-kube-api-access-6m55t\") pod \"payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.184143 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.183962 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2b3ee3-285e-4740-aee2-7003aa50baed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.285642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.285516 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m55t\" (UniqueName: \"kubernetes.io/projected/bd2b3ee3-285e-4740-aee2-7003aa50baed-kube-api-access-6m55t\") pod \"payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.285852 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.285670 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2b3ee3-285e-4740-aee2-7003aa50baed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.286466 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.286442 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2b3ee3-285e-4740-aee2-7003aa50baed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.339417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.339374 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m55t\" (UniqueName: \"kubernetes.io/projected/bd2b3ee3-285e-4740-aee2-7003aa50baed-kube-api-access-6m55t\") pod \"payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.345140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.345089 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:05.735389 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.735358 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct"] Nov 02 19:39:05.738167 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:39:05.738135 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2b3ee3_285e_4740_aee2_7003aa50baed.slice/crio-8f2987d0fa13ee912ffb768189ad625787d6a70b3725615ea60adfb99588c9b4 WatchSource:0}: Error finding container 8f2987d0fa13ee912ffb768189ad625787d6a70b3725615ea60adfb99588c9b4: Status 404 returned error can't find the container with id 8f2987d0fa13ee912ffb768189ad625787d6a70b3725615ea60adfb99588c9b4 Nov 02 19:39:05.930516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:05.930487 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" event={"ID":"bd2b3ee3-285e-4740-aee2-7003aa50baed","Type":"ContainerStarted","Data":"8f2987d0fa13ee912ffb768189ad625787d6a70b3725615ea60adfb99588c9b4"} Nov 02 19:39:06.938783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:06.938677 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" event={"ID":"d7499b11-1af4-4ca5-acef-68a5d6dcde9b","Type":"ContainerStarted","Data":"1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4"} Nov 02 19:39:06.939204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:06.938813 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:06.940359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:06.940331 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" event={"ID":"bd2b3ee3-285e-4740-aee2-7003aa50baed","Type":"ContainerStarted","Data":"15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a"} Nov 02 19:39:06.940494 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:06.940471 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:07.046199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:07.046141 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" podStartSLOduration=2.150501278 podStartE2EDuration="5.046125647s" totalImagesPullingTime="2.895624369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:07.043651803 +0000 UTC m=+652.992307643" watchObservedRunningTime="2026-11-02 19:39:07.046125647 +0000 UTC m=+652.994781486" Nov 02 19:39:07.151359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:07.151291 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" podStartSLOduration=2.365941707 podStartE2EDuration="3.151273581s" totalImagesPullingTime="785.331874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:07.148582368 +0000 UTC m=+653.097238207" watchObservedRunningTime="2026-11-02 19:39:07.151273581 +0000 UTC m=+653.099929419" Nov 02 19:39:10.152837 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.152797 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9"] Nov 02 19:39:10.157032 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.157007 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.206475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.206443 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-26cfbe\"" Nov 02 19:39:10.209615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.209583 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9"] Nov 02 19:39:10.248276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.248235 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c29350-0a6c-48cd-abfd-7d55a3903c34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.248582 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.248559 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44w8z\" (UniqueName: \"kubernetes.io/projected/93c29350-0a6c-48cd-abfd-7d55a3903c34-kube-api-access-44w8z\") pod \"payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.252438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.252419 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-26cfbe-dockercfg-xfmsw\"" Nov 02 19:39:10.349775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.349713 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c29350-0a6c-48cd-abfd-7d55a3903c34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.349775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.349781 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44w8z\" (UniqueName: \"kubernetes.io/projected/93c29350-0a6c-48cd-abfd-7d55a3903c34-kube-api-access-44w8z\") pod \"payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.350644 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.350622 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c29350-0a6c-48cd-abfd-7d55a3903c34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.403369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.403275 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44w8z\" (UniqueName: \"kubernetes.io/projected/93c29350-0a6c-48cd-abfd-7d55a3903c34-kube-api-access-44w8z\") pod \"payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.476293 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.476260 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.843596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.843567 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9"] Nov 02 19:39:10.845265 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:39:10.845237 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c29350_0a6c_48cd_abfd_7d55a3903c34.slice/crio-7b3fcc06132ebf91d560b3a20736e6c7f144ad1d0b890a821a748d29d59965da WatchSource:0}: Error finding container 7b3fcc06132ebf91d560b3a20736e6c7f144ad1d0b890a821a748d29d59965da: Status 404 returned error can't find the container with id 7b3fcc06132ebf91d560b3a20736e6c7f144ad1d0b890a821a748d29d59965da Nov 02 19:39:10.965259 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.965219 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" event={"ID":"93c29350-0a6c-48cd-abfd-7d55a3903c34","Type":"ContainerStarted","Data":"55c348d1e3469e1abd106b4ba06bce663dfaf9ad854712ea7c33567b0d9c09c0"} Nov 02 19:39:10.965467 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.965291 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:10.965467 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:10.965327 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" event={"ID":"93c29350-0a6c-48cd-abfd-7d55a3903c34","Type":"ContainerStarted","Data":"7b3fcc06132ebf91d560b3a20736e6c7f144ad1d0b890a821a748d29d59965da"} Nov 02 19:39:14.751380 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:39:14.751346 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:39:15.183575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.183462 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" podStartSLOduration=5.18344608 podStartE2EDuration="5.18344608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:11.09524359 +0000 UTC m=+657.043899429" watchObservedRunningTime="2026-11-02 19:39:15.18344608 +0000 UTC m=+661.132101915" Nov 02 19:39:15.184910 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.184886 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv"] Nov 02 19:39:15.188824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.188805 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.285742 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.285710 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-e9c144\"" Nov 02 19:39:15.285742 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.285725 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-e9c144-dockercfg-jkx6r\"" Nov 02 19:39:15.288568 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.288541 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv"] Nov 02 19:39:15.310320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.301236 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.310320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.301420 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzds\" (UniqueName: \"kubernetes.io/projected/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-kube-api-access-6qzds\") pod \"payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.334646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.334611 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49"] Nov 02 19:39:15.338447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.338426 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.378617 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.378578 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49"] Nov 02 19:39:15.403039 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.403003 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.403235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.403056 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44bh\" (UniqueName: \"kubernetes.io/projected/3ace283e-959e-4523-b74b-680ad65469bf-kube-api-access-l44bh\") pod \"payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.403235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.403107 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzds\" (UniqueName: \"kubernetes.io/projected/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-kube-api-access-6qzds\") pod \"payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.403235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.403225 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ace283e-959e-4523-b74b-680ad65469bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.404061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.404037 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.455719 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.455653 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzds\" (UniqueName: \"kubernetes.io/projected/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-kube-api-access-6qzds\") pod \"payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.502711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.502686 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:15.504051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.504030 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ace283e-959e-4523-b74b-680ad65469bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.504119 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.504090 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44bh\" (UniqueName: \"kubernetes.io/projected/3ace283e-959e-4523-b74b-680ad65469bf-kube-api-access-l44bh\") pod \"payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.504719 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.504699 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ace283e-959e-4523-b74b-680ad65469bf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.560273 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.557727 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44bh\" (UniqueName: \"kubernetes.io/projected/3ace283e-959e-4523-b74b-680ad65469bf-kube-api-access-l44bh\") pod \"payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.651958 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.651920 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:15.866977 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.866951 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv"] Nov 02 19:39:15.998484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.998453 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" event={"ID":"25a17621-7bdd-49aa-a4bd-5a2d4f98903a","Type":"ContainerStarted","Data":"410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422"} Nov 02 19:39:15.998669 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.998501 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" event={"ID":"25a17621-7bdd-49aa-a4bd-5a2d4f98903a","Type":"ContainerStarted","Data":"865d9e5d4df6f6f5ec2e325cd4c80ce9b1d387c260432c8022e68536682fa2d1"} Nov 02 19:39:15.998669 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:15.998605 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:16.016452 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:16.016431 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49"] Nov 02 19:39:16.018483 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:39:16.018452 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ace283e_959e_4523_b74b_680ad65469bf.slice/crio-1d2363ad29c55cf4ea57ee4ed7517b36e8b6e30358f031c3ed858d5eeffe807d WatchSource:0}: Error finding container 1d2363ad29c55cf4ea57ee4ed7517b36e8b6e30358f031c3ed858d5eeffe807d: Status 404 returned error can't find the container with id 1d2363ad29c55cf4ea57ee4ed7517b36e8b6e30358f031c3ed858d5eeffe807d Nov 02 19:39:16.105565 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:16.105515 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" podStartSLOduration=1.105496498 podStartE2EDuration="1.105496498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:16.103622939 +0000 UTC m=+662.052278788" watchObservedRunningTime="2026-11-02 19:39:16.105496498 +0000 UTC m=+662.054152337" Nov 02 19:39:17.005648 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:17.005614 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" event={"ID":"3ace283e-959e-4523-b74b-680ad65469bf","Type":"ContainerStarted","Data":"b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736"} Nov 02 19:39:17.006149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:17.005665 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" event={"ID":"3ace283e-959e-4523-b74b-680ad65469bf","Type":"ContainerStarted","Data":"1d2363ad29c55cf4ea57ee4ed7517b36e8b6e30358f031c3ed858d5eeffe807d"} Nov 02 19:39:17.116067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:17.116003 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" podStartSLOduration=2.115984073 podStartE2EDuration="2.115984073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:17.11209356 +0000 UTC m=+663.060749402" watchObservedRunningTime="2026-11-02 19:39:17.115984073 +0000 UTC m=+663.064639912" Nov 02 19:39:17.949529 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:17.949496 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 19:39:17.949732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:17.949720 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 19:39:18.011347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:18.011289 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:21.977337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:21.977242 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 19:39:22.937324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:22.930800 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98"] Nov 02 19:39:22.956332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:22.945098 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:22.988076 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:22.988037 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98"] Nov 02 19:39:22.993719 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:22.993691 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-28623b\"" Nov 02 19:39:23.028519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.028481 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkpm\" (UniqueName: \"kubernetes.io/projected/4f1f36d2-2a02-463d-b9ce-73265a4407af-kube-api-access-klkpm\") pod \"payload-processing-e2e-worker-w2-28623b-566849489c-pnr98\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:23.028680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.028558 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1f36d2-2a02-463d-b9ce-73265a4407af-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-28623b-566849489c-pnr98\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:23.031195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.031169 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-28623b-dockercfg-xvvkq\"" Nov 02 19:39:23.130177 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.130136 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1f36d2-2a02-463d-b9ce-73265a4407af-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-28623b-566849489c-pnr98\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:23.130406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.130256 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klkpm\" (UniqueName: \"kubernetes.io/projected/4f1f36d2-2a02-463d-b9ce-73265a4407af-kube-api-access-klkpm\") pod \"payload-processing-e2e-worker-w2-28623b-566849489c-pnr98\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:23.130964 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.130936 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1f36d2-2a02-463d-b9ce-73265a4407af-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-28623b-566849489c-pnr98\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:23.185972 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.185937 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkpm\" (UniqueName: \"kubernetes.io/projected/4f1f36d2-2a02-463d-b9ce-73265a4407af-kube-api-access-klkpm\") pod \"payload-processing-e2e-worker-w2-28623b-566849489c-pnr98\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:23.281489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.281457 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:23.682822 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:23.682793 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98"] Nov 02 19:39:24.084728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:24.084692 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" event={"ID":"4f1f36d2-2a02-463d-b9ce-73265a4407af","Type":"ContainerStarted","Data":"70aeeb92c6761c5a3a373938fc89490e4db0549e26ae9f71f1f45003b85eeca2"} Nov 02 19:39:24.085286 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:24.085262 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" event={"ID":"4f1f36d2-2a02-463d-b9ce-73265a4407af","Type":"ContainerStarted","Data":"8f0fdd4709a1cee5d1b18838cb11108d050a7fd80005936d5cc0a1c96b79843a"} Nov 02 19:39:24.085434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:24.085334 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:24.190868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:24.190802 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" podStartSLOduration=2.19078391 podStartE2EDuration="2.19078391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:24.189254729 +0000 UTC m=+670.137910568" watchObservedRunningTime="2026-11-02 19:39:24.19078391 +0000 UTC m=+670.139439750" Nov 02 19:39:27.009282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:27.009202 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 19:39:29.019625 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:29.019594 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 19:39:32.137119 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:32.137085 4966 generic.go:356] "Generic (PLEG): container finished" podID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerID="e93c9d49d20fd7570dd8464a758368a4f40284ae2ff48912c2740c8d840ed7b9" exitCode=0 Nov 02 19:39:32.137600 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:32.137153 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" event={"ID":"aa43ea8b-4430-49d2-940a-dcd655dbc83e","Type":"ContainerDied","Data":"e93c9d49d20fd7570dd8464a758368a4f40284ae2ff48912c2740c8d840ed7b9"} Nov 02 19:39:32.137676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:32.137619 4966 scope.go:122] "RemoveContainer" containerID="e93c9d49d20fd7570dd8464a758368a4f40284ae2ff48912c2740c8d840ed7b9" Nov 02 19:39:32.139056 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:32.139024 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:39:32.139181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:32.139101 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:39:32.139181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:32.139133 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:39:33.167670 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:33.167635 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" event={"ID":"aa43ea8b-4430-49d2-940a-dcd655dbc83e","Type":"ContainerStarted","Data":"ad6df2f2ee40418ce7b16e1c524773595f9834bbb2565af0c2496d315411171e"} Nov 02 19:39:33.611996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:33.611963 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:39:33.613450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:33.613416 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 19:39:33.613611 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:33.613485 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 19:39:33.824441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:33.824400 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:39:34.612145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.612113 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 19:39:34.612596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.612165 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 19:39:34.865734 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.865626 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc"] Nov 02 19:39:34.873019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.872994 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.888153 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.888122 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe8400-2d35-4658-a087-5a12feb990cc-tls-certs\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.888360 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.888199 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkfk\" (UniqueName: \"kubernetes.io/projected/04fe8400-2d35-4658-a087-5a12feb990cc-kube-api-access-wgkfk\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.888360 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.888315 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-home\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.888514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.888383 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-tmp-dir\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.888514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.888433 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-dshm\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.888514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.888474 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-model-cache\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.908256 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.908223 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc"] Nov 02 19:39:34.962762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.962721 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"openshift-service-ca.crt\"" Nov 02 19:39:34.962762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.962758 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"default-dockercfg-f89sw\"" Nov 02 19:39:34.967807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.967778 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"simulator-w3-5c46af-kserve-self-signed-certs\"" Nov 02 19:39:34.967807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.967790 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"kube-root-ca.crt\"" Nov 02 19:39:34.989971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.989940 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe8400-2d35-4658-a087-5a12feb990cc-tls-certs\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.989999 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkfk\" (UniqueName: \"kubernetes.io/projected/04fe8400-2d35-4658-a087-5a12feb990cc-kube-api-access-wgkfk\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.990038 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-home\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.990067 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-tmp-dir\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.990099 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-dshm\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.990123 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-model-cache\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.990542 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-model-cache\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.990691 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-home\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.990914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.990879 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-tmp-dir\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.993766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.993248 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-dshm\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:34.993766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:34.993267 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe8400-2d35-4658-a087-5a12feb990cc-tls-certs\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:35.047951 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:35.047884 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkfk\" (UniqueName: \"kubernetes.io/projected/04fe8400-2d35-4658-a087-5a12feb990cc-kube-api-access-wgkfk\") pod \"simulator-w3-5c46af-kserve-9db7c84fc-npskc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:35.093639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:35.093609 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 19:39:35.187045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:35.186963 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:35.582401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:35.582361 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc"] Nov 02 19:39:35.584483 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:39:35.584454 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04fe8400_2d35_4658_a087_5a12feb990cc.slice/crio-2a57d757ce49d5ddde0fdb27e8dc6a170dc441e469a8d061eec3d575d3bd0ba3 WatchSource:0}: Error finding container 2a57d757ce49d5ddde0fdb27e8dc6a170dc441e469a8d061eec3d575d3bd0ba3: Status 404 returned error can't find the container with id 2a57d757ce49d5ddde0fdb27e8dc6a170dc441e469a8d061eec3d575d3bd0ba3 Nov 02 19:39:35.612590 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:35.612559 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 19:39:35.613025 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:35.612617 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 19:39:36.191495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:36.191445 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" event={"ID":"04fe8400-2d35-4658-a087-5a12feb990cc","Type":"ContainerStarted","Data":"2a57d757ce49d5ddde0fdb27e8dc6a170dc441e469a8d061eec3d575d3bd0ba3"} Nov 02 19:39:36.613367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:36.611690 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 19:39:36.613367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:36.611755 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 19:39:37.612102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:37.612072 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 02 19:39:37.612225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:37.612134 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 02 19:39:38.216267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:38.214635 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" event={"ID":"04fe8400-2d35-4658-a087-5a12feb990cc","Type":"ContainerStarted","Data":"d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0"} Nov 02 19:39:38.325547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:38.325496 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podStartSLOduration=2.32531859 podStartE2EDuration="4.325481823s" totalImagesPullingTime="2.000163233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:38.32235381 +0000 UTC m=+684.271009650" watchObservedRunningTime="2026-11-02 19:39:38.325481823 +0000 UTC m=+684.274137686" Nov 02 19:39:38.616797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:38.616765 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:39:38.617803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:38.617782 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:39:39.187701 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:39.187661 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:44.774252 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:39:44.774201 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:39:45.187132 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:45.187043 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:45.205967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:45.205938 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:45.289587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:45.289559 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:39:49.832231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:49.832150 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 19:39:49.832676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:49.832239 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 19:39:50.318981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:50.318940 4966 generic.go:356] "Generic (PLEG): container finished" podID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerID="618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b" exitCode=0 Nov 02 19:39:50.318981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:50.318973 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" event={"ID":"c0861dab-4be6-4d9e-b2bb-e7f214200037","Type":"ContainerDied","Data":"618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b"} Nov 02 19:39:50.319424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:50.319406 4966 scope.go:122] "RemoveContainer" containerID="618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b" Nov 02 19:39:50.320688 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:50.320659 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:39:50.320790 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:50.320723 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:39:50.320790 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:50.320756 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:39:51.327726 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:51.327693 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" event={"ID":"c0861dab-4be6-4d9e-b2bb-e7f214200037","Type":"ContainerStarted","Data":"a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa"} Nov 02 19:39:51.608098 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:51.608012 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:39:51.608254 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:51.608206 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 19:39:51.608323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:51.608251 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 19:39:52.608105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:52.608071 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 19:39:52.608520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:52.608122 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 19:39:53.608141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:53.608108 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 19:39:53.608546 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:53.608159 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 19:39:54.608642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:54.608604 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 19:39:54.609036 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:54.608673 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 19:39:55.608463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:55.608423 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 02 19:39:55.608653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:55.608491 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 02 19:39:56.612650 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:56.612619 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:39:56.613094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:56.612900 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:39:56.613743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:56.613718 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 19:39:57.766366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.766321 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw"] Nov 02 19:39:57.770754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.770730 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.837447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.837410 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91074226-875a-4294-80f9-d761498e387a-tls-certs\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.837614 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.837553 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-dshm\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.837614 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.837598 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-model-cache\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.837703 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.837664 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-home\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.837744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.837699 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mpd\" (UniqueName: \"kubernetes.io/projected/91074226-875a-4294-80f9-d761498e387a-kube-api-access-76mpd\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.837801 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.837784 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-tmp-dir\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.869946 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.869908 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"openshift-service-ca.crt\"" Nov 02 19:39:57.870131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.869984 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"simulator-w0-43ceaa-kserve-self-signed-certs\"" Nov 02 19:39:57.870131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.869998 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"default-dockercfg-824k2\"" Nov 02 19:39:57.870131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.869988 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"kube-root-ca.crt\"" Nov 02 19:39:57.874092 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.874066 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw"] Nov 02 19:39:57.939131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.939092 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-tmp-dir\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.939355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.939196 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91074226-875a-4294-80f9-d761498e387a-tls-certs\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.939355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.939261 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-dshm\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.939355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.939285 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-model-cache\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.939355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.939349 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-home\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.939645 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.939377 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mpd\" (UniqueName: \"kubernetes.io/projected/91074226-875a-4294-80f9-d761498e387a-kube-api-access-76mpd\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.940009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.939984 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-tmp-dir\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.940626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.940606 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-model-cache\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.940765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.940740 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-home\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.943132 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.943087 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-dshm\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.969322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.960148 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91074226-875a-4294-80f9-d761498e387a-tls-certs\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:57.993194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:57.993137 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mpd\" (UniqueName: \"kubernetes.io/projected/91074226-875a-4294-80f9-d761498e387a-kube-api-access-76mpd\") pod \"simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:58.083628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:58.083516 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:39:58.467029 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:58.467005 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw"] Nov 02 19:39:58.468910 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:39:58.468879 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91074226_875a_4294_80f9_d761498e387a.slice/crio-b7140f2510ad20abeb7741d31991c3cb96aad74da4d9b836a2843c9fa7e4b97c WatchSource:0}: Error finding container b7140f2510ad20abeb7741d31991c3cb96aad74da4d9b836a2843c9fa7e4b97c: Status 404 returned error can't find the container with id b7140f2510ad20abeb7741d31991c3cb96aad74da4d9b836a2843c9fa7e4b97c Nov 02 19:39:59.402103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:59.402070 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" event={"ID":"91074226-875a-4294-80f9-d761498e387a","Type":"ContainerStarted","Data":"251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a"} Nov 02 19:39:59.402103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:59.402110 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" event={"ID":"91074226-875a-4294-80f9-d761498e387a","Type":"ContainerStarted","Data":"b7140f2510ad20abeb7741d31991c3cb96aad74da4d9b836a2843c9fa7e4b97c"} Nov 02 19:39:59.509063 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:39:59.509011 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podStartSLOduration=2.508996718 podStartE2EDuration="2.508996718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:39:59.507515529 +0000 UTC m=+705.456171368" watchObservedRunningTime="2026-11-02 19:39:59.508996718 +0000 UTC m=+705.457652557" Nov 02 19:40:00.083931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:00.083895 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:40:08.083802 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:08.083762 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:40:08.096612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:08.096583 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:40:08.478506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:08.478472 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:40:14.750516 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:40:14.750477 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:40:16.581537 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.581503 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9"] Nov 02 19:40:16.588445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.588426 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.639602 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.639568 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9"] Nov 02 19:40:16.682084 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.682056 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"kube-root-ca.crt\"" Nov 02 19:40:16.682265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.682223 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"default-dockercfg-h46nk\"" Nov 02 19:40:16.682265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.682226 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"openshift-service-ca.crt\"" Nov 02 19:40:16.683288 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.683273 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8795bf\"/\"simulator-w4-8795bf-kserve-self-signed-certs\"" Nov 02 19:40:16.735924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.735897 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/519e8de1-8721-4152-b336-f9cfcf3ab87c-tls-certs\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.736081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.735925 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtssc\" (UniqueName: \"kubernetes.io/projected/519e8de1-8721-4152-b336-f9cfcf3ab87c-kube-api-access-gtssc\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.736081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.736037 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-tmp-dir\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.736176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.736089 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-dshm\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.736176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.736162 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-model-cache\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.736239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.736213 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-home\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.837388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.837257 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-tmp-dir\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.837388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.837324 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-dshm\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.837388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.837360 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-model-cache\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.837785 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.837760 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-tmp-dir\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.837890 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.837794 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-model-cache\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.837958 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.837934 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-home\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.838050 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.838031 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/519e8de1-8721-4152-b336-f9cfcf3ab87c-tls-certs\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.838125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.838064 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtssc\" (UniqueName: \"kubernetes.io/projected/519e8de1-8721-4152-b336-f9cfcf3ab87c-kube-api-access-gtssc\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.838707 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.838682 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-home\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.843974 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.843936 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/519e8de1-8721-4152-b336-f9cfcf3ab87c-tls-certs\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.845325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.845287 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-dshm\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.891276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.891238 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtssc\" (UniqueName: \"kubernetes.io/projected/519e8de1-8721-4152-b336-f9cfcf3ab87c-kube-api-access-gtssc\") pod \"simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:16.901957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:16.901920 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:17.298366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:17.298330 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9"] Nov 02 19:40:17.299955 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:40:17.299924 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod519e8de1_8721_4152_b336_f9cfcf3ab87c.slice/crio-d7217bca892046f86d858ea1bf7055f9be9766c40ae49507163e409857844ea9 WatchSource:0}: Error finding container d7217bca892046f86d858ea1bf7055f9be9766c40ae49507163e409857844ea9: Status 404 returned error can't find the container with id d7217bca892046f86d858ea1bf7055f9be9766c40ae49507163e409857844ea9 Nov 02 19:40:17.525049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:17.525009 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" event={"ID":"519e8de1-8721-4152-b336-f9cfcf3ab87c","Type":"ContainerStarted","Data":"ab29deba28ccbf6418a0a8344f8fe72a47f5309798562c3e5dcd46e2bedf9a65"} Nov 02 19:40:17.525049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:17.525056 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" event={"ID":"519e8de1-8721-4152-b336-f9cfcf3ab87c","Type":"ContainerStarted","Data":"d7217bca892046f86d858ea1bf7055f9be9766c40ae49507163e409857844ea9"} Nov 02 19:40:17.648531 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:17.648414 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podStartSLOduration=1.64839695 podStartE2EDuration="1.64839695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:40:17.645578481 +0000 UTC m=+723.594234322" watchObservedRunningTime="2026-11-02 19:40:17.64839695 +0000 UTC m=+723.597052788" Nov 02 19:40:17.902836 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:17.902762 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:23.713567 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.713529 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f"] Nov 02 19:40:23.718014 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.717994 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.814287 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.814254 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-tmp-dir\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.814462 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.814427 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-dshm\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.814508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.814475 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-model-cache\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.814543 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.814517 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5c46af\"/\"premium-w3-5c46af-kserve-self-signed-certs\"" Nov 02 19:40:23.814577 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.814544 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlpr9\" (UniqueName: \"kubernetes.io/projected/dce95b19-3999-488e-9653-50055cf66632-kube-api-access-mlpr9\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.814638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.814625 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-home\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.814693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.814681 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce95b19-3999-488e-9653-50055cf66632-tls-certs\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.817185 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.817166 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f"] Nov 02 19:40:23.915524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.915461 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-tmp-dir\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.915524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.915517 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-dshm\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.915778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.915575 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-model-cache\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.915778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.915647 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlpr9\" (UniqueName: \"kubernetes.io/projected/dce95b19-3999-488e-9653-50055cf66632-kube-api-access-mlpr9\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.915897 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.915848 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-home\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.915979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.915948 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-tmp-dir\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.916164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.916138 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-model-cache\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.916263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.916233 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-home\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.916502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.916385 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce95b19-3999-488e-9653-50055cf66632-tls-certs\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.918769 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.918734 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-dshm\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.919535 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.919513 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce95b19-3999-488e-9653-50055cf66632-tls-certs\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:23.975006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:23.974979 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlpr9\" (UniqueName: \"kubernetes.io/projected/dce95b19-3999-488e-9653-50055cf66632-kube-api-access-mlpr9\") pod \"premium-w3-5c46af-kserve-5df64676bd-kss6f\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:24.034162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:24.034129 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:24.438342 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:24.438292 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f"] Nov 02 19:40:24.440689 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:40:24.440657 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddce95b19_3999_488e_9653_50055cf66632.slice/crio-626c024a1d856d3003c939af626d89b42003c9eaeb6cfa87d79137b167b929f0 WatchSource:0}: Error finding container 626c024a1d856d3003c939af626d89b42003c9eaeb6cfa87d79137b167b929f0: Status 404 returned error can't find the container with id 626c024a1d856d3003c939af626d89b42003c9eaeb6cfa87d79137b167b929f0 Nov 02 19:40:24.571354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:24.571291 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" event={"ID":"dce95b19-3999-488e-9653-50055cf66632","Type":"ContainerStarted","Data":"dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd"} Nov 02 19:40:24.571564 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:24.571362 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" event={"ID":"dce95b19-3999-488e-9653-50055cf66632","Type":"ContainerStarted","Data":"626c024a1d856d3003c939af626d89b42003c9eaeb6cfa87d79137b167b929f0"} Nov 02 19:40:24.677805 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:24.677747 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podStartSLOduration=1.677730182 podStartE2EDuration="1.677730182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:40:24.676562764 +0000 UTC m=+730.625218598" watchObservedRunningTime="2026-11-02 19:40:24.677730182 +0000 UTC m=+730.626386021" Nov 02 19:40:25.035070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:25.035032 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:26.903025 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:26.902991 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:26.915537 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:26.915512 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:27.599268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:27.599242 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:40:34.035244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:34.035203 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:34.047868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:34.047843 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:34.642129 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:40:34.642096 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:40:44.763492 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:40:44.763454 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:41:06.217440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.217405 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8"] Nov 02 19:41:06.221391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.221373 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.283380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.283342 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tmp-dir\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.283577 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.283395 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-dshm\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.283669 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.283642 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tls-certs\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.283740 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.283701 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-home\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.283796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.283737 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-model-cache\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.283796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.283776 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpk4g\" (UniqueName: \"kubernetes.io/projected/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-kube-api-access-gpk4g\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.284628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.284606 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"kube-root-ca.crt\"" Nov 02 19:41:06.316354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.316275 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"openshift-service-ca.crt\"" Nov 02 19:41:06.316468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.316287 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"default-dockercfg-tjrwt\"" Nov 02 19:41:06.316468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.316377 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"simulator-w2-28623b-kserve-self-signed-certs\"" Nov 02 19:41:06.319142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.319123 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8"] Nov 02 19:41:06.384783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.384746 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tmp-dir\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.384783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.384794 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-dshm\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.385061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.384879 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tls-certs\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.385061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.384914 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-home\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.385061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.384951 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-model-cache\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.385061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.384990 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpk4g\" (UniqueName: \"kubernetes.io/projected/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-kube-api-access-gpk4g\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.385263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.385230 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tmp-dir\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.385562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.385537 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-home\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.385725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.385652 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-model-cache\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.388068 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.388045 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tls-certs\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.388162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.388141 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-dshm\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.438502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.438477 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpk4g\" (UniqueName: \"kubernetes.io/projected/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-kube-api-access-gpk4g\") pod \"simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.534355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.534316 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:06.916939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:06.916897 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8"] Nov 02 19:41:06.918801 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:41:06.918776 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb698a18c_5871_4d0c_8ca3_387fcee7e0e1.slice/crio-cc75b90fa2cab8374e4259672fe9be7af5006e6376ed64d0b171e42f1b4238f0 WatchSource:0}: Error finding container cc75b90fa2cab8374e4259672fe9be7af5006e6376ed64d0b171e42f1b4238f0: Status 404 returned error can't find the container with id cc75b90fa2cab8374e4259672fe9be7af5006e6376ed64d0b171e42f1b4238f0 Nov 02 19:41:07.856134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:07.856098 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" event={"ID":"b698a18c-5871-4d0c-8ca3-387fcee7e0e1","Type":"ContainerStarted","Data":"692c365279f3cd7ae4037b470ece5c68e04cf86588ff70eaea2d487d9c8fcc91"} Nov 02 19:41:07.856134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:07.856137 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" event={"ID":"b698a18c-5871-4d0c-8ca3-387fcee7e0e1","Type":"ContainerStarted","Data":"cc75b90fa2cab8374e4259672fe9be7af5006e6376ed64d0b171e42f1b4238f0"} Nov 02 19:41:07.995418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:07.995364 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podStartSLOduration=1.9953497740000001 podStartE2EDuration="1.995349774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:41:07.993518278 +0000 UTC m=+773.942174120" watchObservedRunningTime="2026-11-02 19:41:07.995349774 +0000 UTC m=+773.944005612" Nov 02 19:41:08.534636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:08.534600 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:14.761467 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:41:14.761428 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:41:16.534789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:16.534750 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:16.548460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:16.548431 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:16.925809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:41:16.925727 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:41:44.753127 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:41:44.753091 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:42:01.629239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.629202 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5"] Nov 02 19:42:01.634003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.633978 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.685161 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.685129 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"openshift-service-ca.crt\"" Nov 02 19:42:01.689580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.689557 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5"] Nov 02 19:42:01.730700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.730662 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"kube-root-ca.crt\"" Nov 02 19:42:01.730879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.730707 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"default-dockercfg-dnjgm\"" Nov 02 19:42:01.730879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.730787 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"premium-w1-26cfbe-kserve-self-signed-certs\"" Nov 02 19:42:01.776461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.776424 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58bc87bd-519d-4b6e-b6bf-199749452d94-tls-certs\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.776646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.776485 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-tmp-dir\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.776646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.776560 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-home\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.776646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.776588 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpmf\" (UniqueName: \"kubernetes.io/projected/58bc87bd-519d-4b6e-b6bf-199749452d94-kube-api-access-6tpmf\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.776646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.776624 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-dshm\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.776853 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.776692 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-model-cache\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878260 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878205 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-model-cache\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878482 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878310 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58bc87bd-519d-4b6e-b6bf-199749452d94-tls-certs\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878482 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878368 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-tmp-dir\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878691 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-home\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878804 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878717 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-model-cache\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878804 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878749 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpmf\" (UniqueName: \"kubernetes.io/projected/58bc87bd-519d-4b6e-b6bf-199749452d94-kube-api-access-6tpmf\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878912 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878803 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-dshm\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.878964 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878906 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-tmp-dir\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.879022 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.878985 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-home\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.881125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.881058 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-dshm\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.881556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.881530 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58bc87bd-519d-4b6e-b6bf-199749452d94-tls-certs\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.931816 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.931782 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpmf\" (UniqueName: \"kubernetes.io/projected/58bc87bd-519d-4b6e-b6bf-199749452d94-kube-api-access-6tpmf\") pod \"premium-w1-26cfbe-kserve-64468758f-z2bj5\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:01.949412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:01.949378 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:02.351252 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:02.351221 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5"] Nov 02 19:42:02.352381 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:42:02.352347 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58bc87bd_519d_4b6e_b6bf_199749452d94.slice/crio-440e121123fd2b37ade3f18a9130c84f76be6b5f39a75f47bc2aeca5ddddb072 WatchSource:0}: Error finding container 440e121123fd2b37ade3f18a9130c84f76be6b5f39a75f47bc2aeca5ddddb072: Status 404 returned error can't find the container with id 440e121123fd2b37ade3f18a9130c84f76be6b5f39a75f47bc2aeca5ddddb072 Nov 02 19:42:03.217054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:03.217013 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" event={"ID":"58bc87bd-519d-4b6e-b6bf-199749452d94","Type":"ContainerStarted","Data":"6c13f9e0f78b3a3adfdf428b2d564b20d0e4bdf52a58579bce429fb31b5c9942"} Nov 02 19:42:03.217478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:03.217074 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" event={"ID":"58bc87bd-519d-4b6e-b6bf-199749452d94","Type":"ContainerStarted","Data":"440e121123fd2b37ade3f18a9130c84f76be6b5f39a75f47bc2aeca5ddddb072"} Nov 02 19:42:03.323926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:03.323872 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podStartSLOduration=2.323858376 podStartE2EDuration="2.323858376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:03.322063192 +0000 UTC m=+829.270719032" watchObservedRunningTime="2026-11-02 19:42:03.323858376 +0000 UTC m=+829.272514215" Nov 02 19:42:03.949918 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:03.949885 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:06.220176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.220148 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz"] Nov 02 19:42:06.224279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.224261 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.320729 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.320638 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:42:06.320729 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.320659 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:42:06.320729 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.320659 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:42:06.320948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.320732 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:42:06.323915 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.323894 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz"] Nov 02 19:42:06.333910 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.333885 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.334049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.333947 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s29wv\" (UniqueName: \"kubernetes.io/projected/f72e0614-b5ff-445f-9f5b-37c18577ce19-kube-api-access-s29wv\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.334049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.334030 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.434813 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.434775 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s29wv\" (UniqueName: \"kubernetes.io/projected/f72e0614-b5ff-445f-9f5b-37c18577ce19-kube-api-access-s29wv\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.435019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.434860 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.435019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.434981 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.435660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.435638 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.437263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.437240 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.490556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.490529 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s29wv\" (UniqueName: \"kubernetes.io/projected/f72e0614-b5ff-445f-9f5b-37c18577ce19-kube-api-access-s29wv\") pod \"maas-api-revoke-sub-cb4d65c3-8tjxz\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.539714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.539686 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:06.909023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:06.908995 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz"] Nov 02 19:42:06.910912 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:42:06.910879 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72e0614_b5ff_445f_9f5b_37c18577ce19.slice/crio-fa71fbc341b7098e9c83619f34f0b424b27ea0e7bec6e3854d7749cb4736ec78 WatchSource:0}: Error finding container fa71fbc341b7098e9c83619f34f0b424b27ea0e7bec6e3854d7749cb4736ec78: Status 404 returned error can't find the container with id fa71fbc341b7098e9c83619f34f0b424b27ea0e7bec6e3854d7749cb4736ec78 Nov 02 19:42:07.244138 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:07.244105 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" event={"ID":"f72e0614-b5ff-445f-9f5b-37c18577ce19","Type":"ContainerStarted","Data":"fa71fbc341b7098e9c83619f34f0b424b27ea0e7bec6e3854d7749cb4736ec78"} Nov 02 19:42:11.949722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:11.949681 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:11.962672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:11.962637 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:12.295032 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:12.295001 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" event={"ID":"f72e0614-b5ff-445f-9f5b-37c18577ce19","Type":"ContainerStarted","Data":"8e62467113064e12921d934f5a29b6ce25f0ccffb158b89dd350ad579eb5825b"} Nov 02 19:42:12.306429 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:12.306401 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:42:12.402765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:12.402712 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" podStartSLOduration=1.792433826 podStartE2EDuration="6.402699586s" totalImagesPullingTime="4.61026576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:12.399812505 +0000 UTC m=+838.348468344" watchObservedRunningTime="2026-11-02 19:42:12.402699586 +0000 UTC m=+838.351355424" Nov 02 19:42:13.302843 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:13.302799 4966 generic.go:356] "Generic (PLEG): container finished" podID="f72e0614-b5ff-445f-9f5b-37c18577ce19" containerID="8e62467113064e12921d934f5a29b6ce25f0ccffb158b89dd350ad579eb5825b" exitCode=0 Nov 02 19:42:13.303344 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:13.302846 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" event={"ID":"f72e0614-b5ff-445f-9f5b-37c18577ce19","Type":"ContainerDied","Data":"8e62467113064e12921d934f5a29b6ce25f0ccffb158b89dd350ad579eb5825b"} Nov 02 19:42:14.125517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.125485 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm"] Nov 02 19:42:14.129417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.129400 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.226156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.226122 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"kube-root-ca.crt\"" Nov 02 19:42:14.226353 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.226232 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"default-dockercfg-2bf9d\"" Nov 02 19:42:14.229374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.229353 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm"] Nov 02 19:42:14.234079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.234062 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"openshift-service-ca.crt\"" Nov 02 19:42:14.234164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.234077 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"premium-w6-e9c144-kserve-self-signed-certs\"" Nov 02 19:42:14.243105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.242738 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-model-cache\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.243105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.242816 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-dshm\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.243105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.242863 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-home\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.243105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.242939 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tls-certs\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.243105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.242992 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tmp-dir\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.243105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.243034 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4dw\" (UniqueName: \"kubernetes.io/projected/6cb81dd2-012d-4a42-9090-5e204dd5f18a-kube-api-access-wj4dw\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.345109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.344614 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-dshm\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.345720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.345699 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-home\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.345905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.345887 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tls-certs\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.346044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.346029 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tmp-dir\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.346154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.346138 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4dw\" (UniqueName: \"kubernetes.io/projected/6cb81dd2-012d-4a42-9090-5e204dd5f18a-kube-api-access-wj4dw\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.346388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.346369 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-model-cache\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.346831 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.346810 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-model-cache\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.347220 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.347201 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tmp-dir\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.347828 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.347531 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-dshm\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.347955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.346041 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-home\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.349161 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.348888 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tls-certs\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.400596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.400013 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4dw\" (UniqueName: \"kubernetes.io/projected/6cb81dd2-012d-4a42-9090-5e204dd5f18a-kube-api-access-wj4dw\") pod \"premium-w6-e9c144-kserve-677dc85f6b-2kdlm\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.442113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.441887 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:14.515770 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.515496 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:14.653141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.653010 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-openshift-ca\") pod \"f72e0614-b5ff-445f-9f5b-37c18577ce19\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " Nov 02 19:42:14.653141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.653107 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-xks-ca\") pod \"f72e0614-b5ff-445f-9f5b-37c18577ce19\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " Nov 02 19:42:14.653399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.653259 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72e0614-b5ff-445f-9f5b-37c18577ce19-kube-api-access-s29wv\" (UniqueName: \"kubernetes.io/projected/f72e0614-b5ff-445f-9f5b-37c18577ce19-kube-api-access-s29wv\") pod \"f72e0614-b5ff-445f-9f5b-37c18577ce19\" (UID: \"f72e0614-b5ff-445f-9f5b-37c18577ce19\") " Nov 02 19:42:14.654840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.654571 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-openshift-ca" pod "f72e0614-b5ff-445f-9f5b-37c18577ce19" (UID: "f72e0614-b5ff-445f-9f5b-37c18577ce19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:42:14.656758 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.656738 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72e0614-b5ff-445f-9f5b-37c18577ce19-kube-api-access-s29wv" pod "f72e0614-b5ff-445f-9f5b-37c18577ce19" (UID: "f72e0614-b5ff-445f-9f5b-37c18577ce19"). InnerVolumeSpecName "kube-api-access-s29wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:42:14.657458 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.657432 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-xks-ca" pod "f72e0614-b5ff-445f-9f5b-37c18577ce19" (UID: "f72e0614-b5ff-445f-9f5b-37c18577ce19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:42:14.755371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.755334 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s29wv\" (UniqueName: \"kubernetes.io/projected/f72e0614-b5ff-445f-9f5b-37c18577ce19-kube-api-access-s29wv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:14.755371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.755371 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:14.755371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.755383 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72e0614-b5ff-445f-9f5b-37c18577ce19-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:14.757340 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:42:14.757273 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:42:14.861238 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:14.861207 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm"] Nov 02 19:42:14.862127 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:42:14.862098 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cb81dd2_012d_4a42_9090_5e204dd5f18a.slice/crio-a0f0ec8f7b88c4999ac506241bc22665f8a0e32b710bace2eb13c263d1608c26 WatchSource:0}: Error finding container a0f0ec8f7b88c4999ac506241bc22665f8a0e32b710bace2eb13c263d1608c26: Status 404 returned error can't find the container with id a0f0ec8f7b88c4999ac506241bc22665f8a0e32b710bace2eb13c263d1608c26 Nov 02 19:42:15.316905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:15.316860 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" event={"ID":"6cb81dd2-012d-4a42-9090-5e204dd5f18a","Type":"ContainerStarted","Data":"def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741"} Nov 02 19:42:15.317091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:15.316913 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" event={"ID":"6cb81dd2-012d-4a42-9090-5e204dd5f18a","Type":"ContainerStarted","Data":"a0f0ec8f7b88c4999ac506241bc22665f8a0e32b710bace2eb13c263d1608c26"} Nov 02 19:42:15.318242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:15.318219 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" Nov 02 19:42:15.318242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:15.318231 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz" event={"ID":"f72e0614-b5ff-445f-9f5b-37c18577ce19","Type":"ContainerDied","Data":"fa71fbc341b7098e9c83619f34f0b424b27ea0e7bec6e3854d7749cb4736ec78"} Nov 02 19:42:15.318451 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:15.318271 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa71fbc341b7098e9c83619f34f0b424b27ea0e7bec6e3854d7749cb4736ec78" Nov 02 19:42:15.423322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:15.423236 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podStartSLOduration=1.423222216 podStartE2EDuration="1.423222216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:15.422920298 +0000 UTC m=+841.371576138" watchObservedRunningTime="2026-11-02 19:42:15.423222216 +0000 UTC m=+841.371878112" Nov 02 19:42:15.442014 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:15.441975 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:16.354347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.354316 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl"] Nov 02 19:42:16.354821 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.354808 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f72e0614-b5ff-445f-9f5b-37c18577ce19" containerName="revoke-keys" Nov 02 19:42:16.354869 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.354822 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72e0614-b5ff-445f-9f5b-37c18577ce19" containerName="revoke-keys" Nov 02 19:42:16.354916 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.354905 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72e0614-b5ff-445f-9f5b-37c18577ce19" containerName="revoke-keys" Nov 02 19:42:16.358206 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.358188 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.409021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.408984 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:42:16.454955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.454924 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:42:16.454955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.454928 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:42:16.454955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.454957 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:42:16.457961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.457910 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl"] Nov 02 19:42:16.474094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.474068 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.474232 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.474107 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.474309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.474280 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52wn\" (UniqueName: \"kubernetes.io/projected/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-kube-api-access-f52wn\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.575400 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.575369 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.575548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.575411 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.575548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.575437 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52wn\" (UniqueName: \"kubernetes.io/projected/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-kube-api-access-f52wn\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.576141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.576119 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.577893 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.577868 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.628070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.628008 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52wn\" (UniqueName: \"kubernetes.io/projected/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-kube-api-access-f52wn\") pod \"maas-api-revoke-sub-bcfaaf09-6fphl\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:16.672253 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:16.672219 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:17.043956 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:17.043921 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl"] Nov 02 19:42:17.333730 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:17.333654 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" event={"ID":"c24276cd-e447-4f78-a8e7-c9c4f09efb9b","Type":"ContainerStarted","Data":"36e37f095d490478c720f68dcdb775f5404f90106e2747085a60ae5780399a0f"} Nov 02 19:42:18.341214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:18.341117 4966 generic.go:356] "Generic (PLEG): container finished" podID="c24276cd-e447-4f78-a8e7-c9c4f09efb9b" containerID="fd50aa0c0a7522654a86f48f3fbc9af9ee9d966c5f84406a9bf8bd6bd5be73a8" exitCode=0 Nov 02 19:42:18.341608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:18.341207 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" event={"ID":"c24276cd-e447-4f78-a8e7-c9c4f09efb9b","Type":"ContainerDied","Data":"fd50aa0c0a7522654a86f48f3fbc9af9ee9d966c5f84406a9bf8bd6bd5be73a8"} Nov 02 19:42:19.517155 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.517128 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:19.608886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.608794 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-openshift-ca\") pod \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " Nov 02 19:42:19.609062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.608910 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-kube-api-access-f52wn\" (UniqueName: \"kubernetes.io/projected/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-kube-api-access-f52wn\") pod \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " Nov 02 19:42:19.609062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.608987 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-xks-ca\") pod \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\" (UID: \"c24276cd-e447-4f78-a8e7-c9c4f09efb9b\") " Nov 02 19:42:19.609211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.609185 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-openshift-ca" pod "c24276cd-e447-4f78-a8e7-c9c4f09efb9b" (UID: "c24276cd-e447-4f78-a8e7-c9c4f09efb9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:42:19.609395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.609376 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:19.611183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.611158 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-xks-ca" pod "c24276cd-e447-4f78-a8e7-c9c4f09efb9b" (UID: "c24276cd-e447-4f78-a8e7-c9c4f09efb9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:42:19.611326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.611233 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-kube-api-access-f52wn" pod "c24276cd-e447-4f78-a8e7-c9c4f09efb9b" (UID: "c24276cd-e447-4f78-a8e7-c9c4f09efb9b"). InnerVolumeSpecName "kube-api-access-f52wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:42:19.710377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.710342 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:19.710377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:19.710374 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f52wn\" (UniqueName: \"kubernetes.io/projected/c24276cd-e447-4f78-a8e7-c9c4f09efb9b-kube-api-access-f52wn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:20.357191 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:20.357153 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" event={"ID":"c24276cd-e447-4f78-a8e7-c9c4f09efb9b","Type":"ContainerDied","Data":"36e37f095d490478c720f68dcdb775f5404f90106e2747085a60ae5780399a0f"} Nov 02 19:42:20.357419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:20.357214 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e37f095d490478c720f68dcdb775f5404f90106e2747085a60ae5780399a0f" Nov 02 19:42:20.357419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:20.357164 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl" Nov 02 19:42:21.630669 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.630631 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9"] Nov 02 19:42:21.631354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.631328 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c24276cd-e447-4f78-a8e7-c9c4f09efb9b" containerName="revoke-keys" Nov 02 19:42:21.631354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.631352 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24276cd-e447-4f78-a8e7-c9c4f09efb9b" containerName="revoke-keys" Nov 02 19:42:21.631550 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.631471 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24276cd-e447-4f78-a8e7-c9c4f09efb9b" containerName="revoke-keys" Nov 02 19:42:21.636897 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.636871 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.689461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.689429 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9"] Nov 02 19:42:21.708665 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.708626 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:42:21.731307 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.731256 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:42:21.731477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.731332 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:42:21.731477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.731369 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.731477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.731410 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:42:21.731477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.731440 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2ps\" (UniqueName: \"kubernetes.io/projected/8546944b-e89c-4534-a504-3cf4ffc1fb60-kube-api-access-pd2ps\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.731639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.731616 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.832941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.832900 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.833119 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.832999 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2ps\" (UniqueName: \"kubernetes.io/projected/8546944b-e89c-4534-a504-3cf4ffc1fb60-kube-api-access-pd2ps\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.833165 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.833119 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.833719 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.833686 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.835586 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.835558 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.886718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.886645 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2ps\" (UniqueName: \"kubernetes.io/projected/8546944b-e89c-4534-a504-3cf4ffc1fb60-kube-api-access-pd2ps\") pod \"maas-api-revoke-sub-1116f601-gz6h9\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:21.954562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:21.954526 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:22.325209 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:22.325180 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9"] Nov 02 19:42:22.326190 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:42:22.326167 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8546944b_e89c_4534_a504_3cf4ffc1fb60.slice/crio-b3ec61a5080ea45ad9709daadd033767bf841359de31e0d6db7ef5f86d53d904 WatchSource:0}: Error finding container b3ec61a5080ea45ad9709daadd033767bf841359de31e0d6db7ef5f86d53d904: Status 404 returned error can't find the container with id b3ec61a5080ea45ad9709daadd033767bf841359de31e0d6db7ef5f86d53d904 Nov 02 19:42:22.370824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:22.370793 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" event={"ID":"8546944b-e89c-4534-a504-3cf4ffc1fb60","Type":"ContainerStarted","Data":"b3ec61a5080ea45ad9709daadd033767bf841359de31e0d6db7ef5f86d53d904"} Nov 02 19:42:23.379288 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:23.379249 4966 generic.go:356] "Generic (PLEG): container finished" podID="8546944b-e89c-4534-a504-3cf4ffc1fb60" containerID="09c958240457da38b609535b470935d5b18003c3281777025bf1b354a7670c13" exitCode=0 Nov 02 19:42:23.379712 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:23.379285 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" event={"ID":"8546944b-e89c-4534-a504-3cf4ffc1fb60","Type":"ContainerDied","Data":"09c958240457da38b609535b470935d5b18003c3281777025bf1b354a7670c13"} Nov 02 19:42:24.443445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.442897 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:24.463318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.463085 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:24.546459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.546433 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:24.660653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.660576 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-xks-ca\") pod \"8546944b-e89c-4534-a504-3cf4ffc1fb60\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " Nov 02 19:42:24.660866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.660681 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-openshift-ca\") pod \"8546944b-e89c-4534-a504-3cf4ffc1fb60\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " Nov 02 19:42:24.660866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.660786 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8546944b-e89c-4534-a504-3cf4ffc1fb60-kube-api-access-pd2ps\" (UniqueName: \"kubernetes.io/projected/8546944b-e89c-4534-a504-3cf4ffc1fb60-kube-api-access-pd2ps\") pod \"8546944b-e89c-4534-a504-3cf4ffc1fb60\" (UID: \"8546944b-e89c-4534-a504-3cf4ffc1fb60\") " Nov 02 19:42:24.661142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.661113 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-openshift-ca" pod "8546944b-e89c-4534-a504-3cf4ffc1fb60" (UID: "8546944b-e89c-4534-a504-3cf4ffc1fb60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:42:24.661263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.661245 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:24.662877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.662846 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-xks-ca" pod "8546944b-e89c-4534-a504-3cf4ffc1fb60" (UID: "8546944b-e89c-4534-a504-3cf4ffc1fb60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:42:24.662994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.662932 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8546944b-e89c-4534-a504-3cf4ffc1fb60-kube-api-access-pd2ps" pod "8546944b-e89c-4534-a504-3cf4ffc1fb60" (UID: "8546944b-e89c-4534-a504-3cf4ffc1fb60"). InnerVolumeSpecName "kube-api-access-pd2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:42:24.762225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.762153 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2ps\" (UniqueName: \"kubernetes.io/projected/8546944b-e89c-4534-a504-3cf4ffc1fb60-kube-api-access-pd2ps\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:24.762225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:24.762196 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8546944b-e89c-4534-a504-3cf4ffc1fb60-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:25.400427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:25.400396 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" Nov 02 19:42:25.400596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:25.400426 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9" event={"ID":"8546944b-e89c-4534-a504-3cf4ffc1fb60","Type":"ContainerDied","Data":"b3ec61a5080ea45ad9709daadd033767bf841359de31e0d6db7ef5f86d53d904"} Nov 02 19:42:25.400596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:25.400500 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3ec61a5080ea45ad9709daadd033767bf841359de31e0d6db7ef5f86d53d904" Nov 02 19:42:25.412090 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:25.412066 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:42:36.282952 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.282917 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d"] Nov 02 19:42:36.283468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.283454 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8546944b-e89c-4534-a504-3cf4ffc1fb60" containerName="revoke-keys" Nov 02 19:42:36.283515 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.283470 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8546944b-e89c-4534-a504-3cf4ffc1fb60" containerName="revoke-keys" Nov 02 19:42:36.283574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.283563 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8546944b-e89c-4534-a504-3cf4ffc1fb60" containerName="revoke-keys" Nov 02 19:42:36.287250 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.287234 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.383510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.383475 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:42:36.383510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.383489 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:42:36.383720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.383514 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:42:36.383720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.383620 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:42:36.386896 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.386870 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d"] Nov 02 19:42:36.423035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.422978 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.423226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.423165 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.423321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.423263 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f6v\" (UniqueName: \"kubernetes.io/projected/c18bd168-3fee-47b4-b464-ee294f996274-kube-api-access-w5f6v\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.524365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.524326 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.524547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.524448 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.524547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.524524 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f6v\" (UniqueName: \"kubernetes.io/projected/c18bd168-3fee-47b4-b464-ee294f996274-kube-api-access-w5f6v\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.525189 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.525165 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.527147 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.527116 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.580352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.580262 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f6v\" (UniqueName: \"kubernetes.io/projected/c18bd168-3fee-47b4-b464-ee294f996274-kube-api-access-w5f6v\") pod \"maas-api-revoke-sub-94fc97cf-9pd9d\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.602518 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.602491 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:36.980535 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:36.980496 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d"] Nov 02 19:42:36.981951 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:42:36.981906 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18bd168_3fee_47b4_b464_ee294f996274.slice/crio-ceb88597fcdb8635a5e6a389ffec9aebb5dc17b358d22eea66c43b6da761a0f6 WatchSource:0}: Error finding container ceb88597fcdb8635a5e6a389ffec9aebb5dc17b358d22eea66c43b6da761a0f6: Status 404 returned error can't find the container with id ceb88597fcdb8635a5e6a389ffec9aebb5dc17b358d22eea66c43b6da761a0f6 Nov 02 19:42:37.480723 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:37.480688 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" event={"ID":"c18bd168-3fee-47b4-b464-ee294f996274","Type":"ContainerStarted","Data":"e777da9dbace88475f623b198aedf4fb673daa8b72fc7a48d154944bdbcdc607"} Nov 02 19:42:37.481116 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:37.480740 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" event={"ID":"c18bd168-3fee-47b4-b464-ee294f996274","Type":"ContainerStarted","Data":"ceb88597fcdb8635a5e6a389ffec9aebb5dc17b358d22eea66c43b6da761a0f6"} Nov 02 19:42:37.586795 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:37.586677 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" podStartSLOduration=1.586660819 podStartE2EDuration="1.586660819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:37.58612758 +0000 UTC m=+863.534783435" watchObservedRunningTime="2026-11-02 19:42:37.586660819 +0000 UTC m=+863.535316661" Nov 02 19:42:38.489647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:38.489610 4966 generic.go:356] "Generic (PLEG): container finished" podID="c18bd168-3fee-47b4-b464-ee294f996274" containerID="e777da9dbace88475f623b198aedf4fb673daa8b72fc7a48d154944bdbcdc607" exitCode=0 Nov 02 19:42:38.490084 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:38.489702 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" event={"ID":"c18bd168-3fee-47b4-b464-ee294f996274","Type":"ContainerDied","Data":"e777da9dbace88475f623b198aedf4fb673daa8b72fc7a48d154944bdbcdc607"} Nov 02 19:42:39.666528 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.666496 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:39.688519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.688483 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18bd168-3fee-47b4-b464-ee294f996274-kube-api-access-w5f6v\" (UniqueName: \"kubernetes.io/projected/c18bd168-3fee-47b4-b464-ee294f996274-kube-api-access-w5f6v\") pod \"c18bd168-3fee-47b4-b464-ee294f996274\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " Nov 02 19:42:39.688697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.688604 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-xks-ca\") pod \"c18bd168-3fee-47b4-b464-ee294f996274\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " Nov 02 19:42:39.690634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.690602 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-xks-ca" pod "c18bd168-3fee-47b4-b464-ee294f996274" (UID: "c18bd168-3fee-47b4-b464-ee294f996274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:42:39.690764 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.690746 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18bd168-3fee-47b4-b464-ee294f996274-kube-api-access-w5f6v" pod "c18bd168-3fee-47b4-b464-ee294f996274" (UID: "c18bd168-3fee-47b4-b464-ee294f996274"). InnerVolumeSpecName "kube-api-access-w5f6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:42:39.790588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.789714 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-openshift-ca\") pod \"c18bd168-3fee-47b4-b464-ee294f996274\" (UID: \"c18bd168-3fee-47b4-b464-ee294f996274\") " Nov 02 19:42:39.790588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.790156 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f6v\" (UniqueName: \"kubernetes.io/projected/c18bd168-3fee-47b4-b464-ee294f996274-kube-api-access-w5f6v\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:39.790588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.790182 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:39.790588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.790175 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-openshift-ca" pod "c18bd168-3fee-47b4-b464-ee294f996274" (UID: "c18bd168-3fee-47b4-b464-ee294f996274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:42:39.891074 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:39.891027 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18bd168-3fee-47b4-b464-ee294f996274-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:40.508270 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:40.508239 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" Nov 02 19:42:40.508590 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:40.508266 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d" event={"ID":"c18bd168-3fee-47b4-b464-ee294f996274","Type":"ContainerDied","Data":"ceb88597fcdb8635a5e6a389ffec9aebb5dc17b358d22eea66c43b6da761a0f6"} Nov 02 19:42:40.508701 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:40.508622 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceb88597fcdb8635a5e6a389ffec9aebb5dc17b358d22eea66c43b6da761a0f6" Nov 02 19:42:44.755285 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:42:44.755246 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:42:47.029514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.029477 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj"] Nov 02 19:42:47.030188 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.030167 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c18bd168-3fee-47b4-b464-ee294f996274" containerName="revoke-keys" Nov 02 19:42:47.030313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.030189 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18bd168-3fee-47b4-b464-ee294f996274" containerName="revoke-keys" Nov 02 19:42:47.030381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.030359 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18bd168-3fee-47b4-b464-ee294f996274" containerName="revoke-keys" Nov 02 19:42:47.038508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.038481 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.088589 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.088559 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj"] Nov 02 19:42:47.095744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.095715 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b44n\" (UniqueName: \"kubernetes.io/projected/89dab0b7-2add-4566-b462-ea938eb712f2-kube-api-access-4b44n\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.095913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.095893 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.095963 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.095933 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.131018 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.130980 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:42:47.131164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.131102 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:42:47.131164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.131158 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:42:47.133045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.133018 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:42:47.197489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.197454 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.197489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.197492 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.197728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.197561 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b44n\" (UniqueName: \"kubernetes.io/projected/89dab0b7-2add-4566-b462-ea938eb712f2-kube-api-access-4b44n\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.198108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.198077 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.199815 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.199790 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.250831 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.250801 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b44n\" (UniqueName: \"kubernetes.io/projected/89dab0b7-2add-4566-b462-ea938eb712f2-kube-api-access-4b44n\") pod \"maas-api-revoke-sub-7534a3f7-c2lgj\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.355444 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.355341 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:47.748572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:47.748541 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj"] Nov 02 19:42:47.750500 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:42:47.750462 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dab0b7_2add_4566_b462_ea938eb712f2.slice/crio-a48f91abcc7ec4d2ed58ef3337f82a3d51d3ab106fc875b74d12a2e951adc8de WatchSource:0}: Error finding container a48f91abcc7ec4d2ed58ef3337f82a3d51d3ab106fc875b74d12a2e951adc8de: Status 404 returned error can't find the container with id a48f91abcc7ec4d2ed58ef3337f82a3d51d3ab106fc875b74d12a2e951adc8de Nov 02 19:42:48.577682 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:48.577635 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" event={"ID":"89dab0b7-2add-4566-b462-ea938eb712f2","Type":"ContainerStarted","Data":"e3ecfc0e459d4501b0ff318bbc0a34f1df64dfa1fc4926cbd85330499519a9ad"} Nov 02 19:42:48.578355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:48.577740 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" event={"ID":"89dab0b7-2add-4566-b462-ea938eb712f2","Type":"ContainerStarted","Data":"a48f91abcc7ec4d2ed58ef3337f82a3d51d3ab106fc875b74d12a2e951adc8de"} Nov 02 19:42:48.683268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:48.683195 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" podStartSLOduration=2.683177693 podStartE2EDuration="2.683177693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:42:48.68279939 +0000 UTC m=+874.631455229" watchObservedRunningTime="2026-11-02 19:42:48.683177693 +0000 UTC m=+874.631833534" Nov 02 19:42:49.586348 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:49.586289 4966 generic.go:356] "Generic (PLEG): container finished" podID="89dab0b7-2add-4566-b462-ea938eb712f2" containerID="e3ecfc0e459d4501b0ff318bbc0a34f1df64dfa1fc4926cbd85330499519a9ad" exitCode=0 Nov 02 19:42:49.586738 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:49.586379 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" event={"ID":"89dab0b7-2add-4566-b462-ea938eb712f2","Type":"ContainerDied","Data":"e3ecfc0e459d4501b0ff318bbc0a34f1df64dfa1fc4926cbd85330499519a9ad"} Nov 02 19:42:50.743033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.743008 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:50.841724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.841694 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-xks-ca\") pod \"89dab0b7-2add-4566-b462-ea938eb712f2\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " Nov 02 19:42:50.841894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.841749 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dab0b7-2add-4566-b462-ea938eb712f2-kube-api-access-4b44n\" (UniqueName: \"kubernetes.io/projected/89dab0b7-2add-4566-b462-ea938eb712f2-kube-api-access-4b44n\") pod \"89dab0b7-2add-4566-b462-ea938eb712f2\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " Nov 02 19:42:50.841894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.841786 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-openshift-ca\") pod \"89dab0b7-2add-4566-b462-ea938eb712f2\" (UID: \"89dab0b7-2add-4566-b462-ea938eb712f2\") " Nov 02 19:42:50.842203 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.842177 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-openshift-ca" pod "89dab0b7-2add-4566-b462-ea938eb712f2" (UID: "89dab0b7-2add-4566-b462-ea938eb712f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:42:50.843820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.843799 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-xks-ca" pod "89dab0b7-2add-4566-b462-ea938eb712f2" (UID: "89dab0b7-2add-4566-b462-ea938eb712f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:42:50.843892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.843865 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dab0b7-2add-4566-b462-ea938eb712f2-kube-api-access-4b44n" pod "89dab0b7-2add-4566-b462-ea938eb712f2" (UID: "89dab0b7-2add-4566-b462-ea938eb712f2"). InnerVolumeSpecName "kube-api-access-4b44n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:42:50.943775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.943739 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b44n\" (UniqueName: \"kubernetes.io/projected/89dab0b7-2add-4566-b462-ea938eb712f2-kube-api-access-4b44n\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:50.943775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.943771 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:50.943775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:50.943781 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dab0b7-2add-4566-b462-ea938eb712f2-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:42:51.601235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:51.601208 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" Nov 02 19:42:51.601399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:51.601240 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj" event={"ID":"89dab0b7-2add-4566-b462-ea938eb712f2","Type":"ContainerDied","Data":"a48f91abcc7ec4d2ed58ef3337f82a3d51d3ab106fc875b74d12a2e951adc8de"} Nov 02 19:42:51.601399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:42:51.601289 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48f91abcc7ec4d2ed58ef3337f82a3d51d3ab106fc875b74d12a2e951adc8de" Nov 02 19:43:01.167759 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.167719 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd"] Nov 02 19:43:01.168481 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.168459 4966 cpu_manager.go:414] "containerMap: removing container" podUID="89dab0b7-2add-4566-b462-ea938eb712f2" containerName="revoke-keys" Nov 02 19:43:01.168534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.168484 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dab0b7-2add-4566-b462-ea938eb712f2" containerName="revoke-keys" Nov 02 19:43:01.168683 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.168671 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dab0b7-2add-4566-b462-ea938eb712f2" containerName="revoke-keys" Nov 02 19:43:01.172645 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.172624 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.223762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.223725 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd"] Nov 02 19:43:01.266414 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.266372 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:01.266414 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.266374 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:01.266650 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.266380 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:01.266832 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.266796 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.266956 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.266854 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.267050 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.267030 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2tf\" (UniqueName: \"kubernetes.io/projected/4b4167e9-46f7-4dd4-b94e-111429636de1-kube-api-access-5c2tf\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.267107 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.267087 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:01.367879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.367842 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.367879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.367881 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.368159 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.368036 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2tf\" (UniqueName: \"kubernetes.io/projected/4b4167e9-46f7-4dd4-b94e-111429636de1-kube-api-access-5c2tf\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.368780 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.368750 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.370667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.370643 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.421183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.421120 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2tf\" (UniqueName: \"kubernetes.io/projected/4b4167e9-46f7-4dd4-b94e-111429636de1-kube-api-access-5c2tf\") pod \"maas-api-revoke-sub-2f021618-khrnd\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.490726 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.490694 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:01.863031 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:01.862995 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4167e9_46f7_4dd4_b94e_111429636de1.slice/crio-cdfabd32a53c6cad18caa6eeda23e409363ff38756ad2865ca5d529e3c96360a WatchSource:0}: Error finding container cdfabd32a53c6cad18caa6eeda23e409363ff38756ad2865ca5d529e3c96360a: Status 404 returned error can't find the container with id cdfabd32a53c6cad18caa6eeda23e409363ff38756ad2865ca5d529e3c96360a Nov 02 19:43:01.868642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:01.868613 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd"] Nov 02 19:43:02.675052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:02.675008 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" event={"ID":"4b4167e9-46f7-4dd4-b94e-111429636de1","Type":"ContainerStarted","Data":"bc79a2ee7dc7764d63129defd514ffe3e87f7d73bbfbd1b400649d3b8fd5399e"} Nov 02 19:43:02.675547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:02.675067 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" event={"ID":"4b4167e9-46f7-4dd4-b94e-111429636de1","Type":"ContainerStarted","Data":"cdfabd32a53c6cad18caa6eeda23e409363ff38756ad2865ca5d529e3c96360a"} Nov 02 19:43:02.781144 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:02.781090 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" podStartSLOduration=1.781071516 podStartE2EDuration="1.781071516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:02.779969148 +0000 UTC m=+888.728624986" watchObservedRunningTime="2026-11-02 19:43:02.781071516 +0000 UTC m=+888.729727385" Nov 02 19:43:03.685330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:03.685269 4966 generic.go:356] "Generic (PLEG): container finished" podID="4b4167e9-46f7-4dd4-b94e-111429636de1" containerID="bc79a2ee7dc7764d63129defd514ffe3e87f7d73bbfbd1b400649d3b8fd5399e" exitCode=0 Nov 02 19:43:03.685761 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:03.685337 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" event={"ID":"4b4167e9-46f7-4dd4-b94e-111429636de1","Type":"ContainerDied","Data":"bc79a2ee7dc7764d63129defd514ffe3e87f7d73bbfbd1b400649d3b8fd5399e"} Nov 02 19:43:05.016352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.014895 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:05.024091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.024054 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-xks-ca\") pod \"4b4167e9-46f7-4dd4-b94e-111429636de1\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " Nov 02 19:43:05.027703 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.027670 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-xks-ca" pod "4b4167e9-46f7-4dd4-b94e-111429636de1" (UID: "4b4167e9-46f7-4dd4-b94e-111429636de1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:05.125642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.125613 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-openshift-ca\") pod \"4b4167e9-46f7-4dd4-b94e-111429636de1\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " Nov 02 19:43:05.125813 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.125654 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4167e9-46f7-4dd4-b94e-111429636de1-kube-api-access-5c2tf\" (UniqueName: \"kubernetes.io/projected/4b4167e9-46f7-4dd4-b94e-111429636de1-kube-api-access-5c2tf\") pod \"4b4167e9-46f7-4dd4-b94e-111429636de1\" (UID: \"4b4167e9-46f7-4dd4-b94e-111429636de1\") " Nov 02 19:43:05.126115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.126079 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:05.126693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.126672 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-openshift-ca" pod "4b4167e9-46f7-4dd4-b94e-111429636de1" (UID: "4b4167e9-46f7-4dd4-b94e-111429636de1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:05.128834 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.128803 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4167e9-46f7-4dd4-b94e-111429636de1-kube-api-access-5c2tf" pod "4b4167e9-46f7-4dd4-b94e-111429636de1" (UID: "4b4167e9-46f7-4dd4-b94e-111429636de1"). InnerVolumeSpecName "kube-api-access-5c2tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:05.228160 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.228049 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4167e9-46f7-4dd4-b94e-111429636de1-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:05.228160 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.228089 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c2tf\" (UniqueName: \"kubernetes.io/projected/4b4167e9-46f7-4dd4-b94e-111429636de1-kube-api-access-5c2tf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:05.704422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.704329 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" Nov 02 19:43:05.704422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.704350 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd" event={"ID":"4b4167e9-46f7-4dd4-b94e-111429636de1","Type":"ContainerDied","Data":"cdfabd32a53c6cad18caa6eeda23e409363ff38756ad2865ca5d529e3c96360a"} Nov 02 19:43:05.704422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:05.704416 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdfabd32a53c6cad18caa6eeda23e409363ff38756ad2865ca5d529e3c96360a" Nov 02 19:43:06.980368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:06.980331 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn"] Nov 02 19:43:06.980925 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:06.980887 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4b4167e9-46f7-4dd4-b94e-111429636de1" containerName="revoke-keys" Nov 02 19:43:06.980925 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:06.980902 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4167e9-46f7-4dd4-b94e-111429636de1" containerName="revoke-keys" Nov 02 19:43:06.981051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:06.980998 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4167e9-46f7-4dd4-b94e-111429636de1" containerName="revoke-keys" Nov 02 19:43:06.984659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:06.984635 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.039688 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.039655 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn"] Nov 02 19:43:07.053586 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.053551 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdqk\" (UniqueName: \"kubernetes.io/projected/b67582cc-de5b-42a8-8bc0-46e5f59614b5-kube-api-access-nwdqk\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.053586 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.053587 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.053830 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.053743 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.091708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.091676 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:07.092750 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.091993 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:07.093040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.092075 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:07.093162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.092170 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:07.155122 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.155085 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdqk\" (UniqueName: \"kubernetes.io/projected/b67582cc-de5b-42a8-8bc0-46e5f59614b5-kube-api-access-nwdqk\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.155122 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.155127 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.155744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.155174 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.156059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.156037 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.157498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.157471 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.212558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.212524 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdqk\" (UniqueName: \"kubernetes.io/projected/b67582cc-de5b-42a8-8bc0-46e5f59614b5-kube-api-access-nwdqk\") pod \"maas-api-revoke-sub-c6403179-69txn\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.299519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.299435 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:07.689354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.689322 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn"] Nov 02 19:43:07.690784 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:07.690753 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67582cc_de5b_42a8_8bc0_46e5f59614b5.slice/crio-b90b80b82b01de68b2e0b188b0627600ad9a6339349ee242d1d0c1d6118045ab WatchSource:0}: Error finding container b90b80b82b01de68b2e0b188b0627600ad9a6339349ee242d1d0c1d6118045ab: Status 404 returned error can't find the container with id b90b80b82b01de68b2e0b188b0627600ad9a6339349ee242d1d0c1d6118045ab Nov 02 19:43:07.722724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:07.722685 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" event={"ID":"b67582cc-de5b-42a8-8bc0-46e5f59614b5","Type":"ContainerStarted","Data":"b90b80b82b01de68b2e0b188b0627600ad9a6339349ee242d1d0c1d6118045ab"} Nov 02 19:43:08.731764 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:08.731724 4966 generic.go:356] "Generic (PLEG): container finished" podID="b67582cc-de5b-42a8-8bc0-46e5f59614b5" containerID="ac3ade01ad55587bdc9cd733dab008814b5a2ca4f8e5e37f540b04ebb71a8630" exitCode=0 Nov 02 19:43:08.732210 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:08.731904 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" event={"ID":"b67582cc-de5b-42a8-8bc0-46e5f59614b5","Type":"ContainerDied","Data":"ac3ade01ad55587bdc9cd733dab008814b5a2ca4f8e5e37f540b04ebb71a8630"} Nov 02 19:43:09.883064 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:09.883037 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:09.992060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:09.991967 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-openshift-ca\") pod \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " Nov 02 19:43:09.992060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:09.992042 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67582cc-de5b-42a8-8bc0-46e5f59614b5-kube-api-access-nwdqk\" (UniqueName: \"kubernetes.io/projected/b67582cc-de5b-42a8-8bc0-46e5f59614b5-kube-api-access-nwdqk\") pod \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " Nov 02 19:43:09.992349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:09.992110 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-xks-ca\") pod \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\" (UID: \"b67582cc-de5b-42a8-8bc0-46e5f59614b5\") " Nov 02 19:43:09.992807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:09.992773 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-openshift-ca" pod "b67582cc-de5b-42a8-8bc0-46e5f59614b5" (UID: "b67582cc-de5b-42a8-8bc0-46e5f59614b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:09.994792 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:09.994761 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67582cc-de5b-42a8-8bc0-46e5f59614b5-kube-api-access-nwdqk" pod "b67582cc-de5b-42a8-8bc0-46e5f59614b5" (UID: "b67582cc-de5b-42a8-8bc0-46e5f59614b5"). InnerVolumeSpecName "kube-api-access-nwdqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:09.995110 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:09.995080 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-xks-ca" pod "b67582cc-de5b-42a8-8bc0-46e5f59614b5" (UID: "b67582cc-de5b-42a8-8bc0-46e5f59614b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:10.094103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:10.094060 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:10.094103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:10.094104 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67582cc-de5b-42a8-8bc0-46e5f59614b5-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:10.094374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:10.094121 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdqk\" (UniqueName: \"kubernetes.io/projected/b67582cc-de5b-42a8-8bc0-46e5f59614b5-kube-api-access-nwdqk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:10.748847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:10.748818 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" Nov 02 19:43:10.748847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:10.748829 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn" event={"ID":"b67582cc-de5b-42a8-8bc0-46e5f59614b5","Type":"ContainerDied","Data":"b90b80b82b01de68b2e0b188b0627600ad9a6339349ee242d1d0c1d6118045ab"} Nov 02 19:43:10.749051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:10.748886 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90b80b82b01de68b2e0b188b0627600ad9a6339349ee242d1d0c1d6118045ab" Nov 02 19:43:12.776594 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.776556 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6"] Nov 02 19:43:12.777198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.777177 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b67582cc-de5b-42a8-8bc0-46e5f59614b5" containerName="revoke-keys" Nov 02 19:43:12.777266 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.777200 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67582cc-de5b-42a8-8bc0-46e5f59614b5" containerName="revoke-keys" Nov 02 19:43:12.777382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.777367 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67582cc-de5b-42a8-8bc0-46e5f59614b5" containerName="revoke-keys" Nov 02 19:43:12.781491 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.781464 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.830414 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.830375 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:12.843538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.843506 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6lw\" (UniqueName: \"kubernetes.io/projected/6d801efa-5982-4805-a41d-cf64ed79f1e3-kube-api-access-9g6lw\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.843678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.843595 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.843678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.843670 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.878711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.877743 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:12.878711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.877889 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:12.878711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.877939 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:12.883696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.883667 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6"] Nov 02 19:43:12.944612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.944573 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.944787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.944706 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6lw\" (UniqueName: \"kubernetes.io/projected/6d801efa-5982-4805-a41d-cf64ed79f1e3-kube-api-access-9g6lw\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.944787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.944779 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.945708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.945677 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.947225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.947200 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:12.998754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:12.998713 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6lw\" (UniqueName: \"kubernetes.io/projected/6d801efa-5982-4805-a41d-cf64ed79f1e3-kube-api-access-9g6lw\") pod \"maas-api-revoke-sub-42ca2b5c-w6jf6\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:13.098666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:13.098578 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:13.480855 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:13.480824 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6"] Nov 02 19:43:13.481380 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:13.481347 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d801efa_5982_4805_a41d_cf64ed79f1e3.slice/crio-08af4f02b3a4a286d284bb01ad6c8a4f7a5bf224aa6f84574e30dd4695d3ad3a WatchSource:0}: Error finding container 08af4f02b3a4a286d284bb01ad6c8a4f7a5bf224aa6f84574e30dd4695d3ad3a: Status 404 returned error can't find the container with id 08af4f02b3a4a286d284bb01ad6c8a4f7a5bf224aa6f84574e30dd4695d3ad3a Nov 02 19:43:13.772312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:13.772268 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" event={"ID":"6d801efa-5982-4805-a41d-cf64ed79f1e3","Type":"ContainerStarted","Data":"08af4f02b3a4a286d284bb01ad6c8a4f7a5bf224aa6f84574e30dd4695d3ad3a"} Nov 02 19:43:14.299862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:14.299836 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:43:14.300521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:14.300502 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:43:14.314726 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:14.314699 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:43:14.315183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:14.315162 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:43:14.754156 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:43:14.754114 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:43:14.781309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:14.781270 4966 generic.go:356] "Generic (PLEG): container finished" podID="6d801efa-5982-4805-a41d-cf64ed79f1e3" containerID="57a7ae8712faa8e0c6d1ac5b962d21d932845af5f52e76e32c4ff31c2cd53c94" exitCode=0 Nov 02 19:43:14.781538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:14.781349 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" event={"ID":"6d801efa-5982-4805-a41d-cf64ed79f1e3","Type":"ContainerDied","Data":"57a7ae8712faa8e0c6d1ac5b962d21d932845af5f52e76e32c4ff31c2cd53c94"} Nov 02 19:43:15.928598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:15.928564 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:16.078623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.078543 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d801efa-5982-4805-a41d-cf64ed79f1e3-kube-api-access-9g6lw\" (UniqueName: \"kubernetes.io/projected/6d801efa-5982-4805-a41d-cf64ed79f1e3-kube-api-access-9g6lw\") pod \"6d801efa-5982-4805-a41d-cf64ed79f1e3\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " Nov 02 19:43:16.078623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.078588 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-openshift-ca\") pod \"6d801efa-5982-4805-a41d-cf64ed79f1e3\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " Nov 02 19:43:16.078836 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.078737 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-xks-ca\") pod \"6d801efa-5982-4805-a41d-cf64ed79f1e3\" (UID: \"6d801efa-5982-4805-a41d-cf64ed79f1e3\") " Nov 02 19:43:16.078988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.078953 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-openshift-ca" pod "6d801efa-5982-4805-a41d-cf64ed79f1e3" (UID: "6d801efa-5982-4805-a41d-cf64ed79f1e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:16.079256 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.079237 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:16.080725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.080702 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d801efa-5982-4805-a41d-cf64ed79f1e3-kube-api-access-9g6lw" pod "6d801efa-5982-4805-a41d-cf64ed79f1e3" (UID: "6d801efa-5982-4805-a41d-cf64ed79f1e3"). InnerVolumeSpecName "kube-api-access-9g6lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:16.080800 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.080729 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-xks-ca" pod "6d801efa-5982-4805-a41d-cf64ed79f1e3" (UID: "6d801efa-5982-4805-a41d-cf64ed79f1e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:16.180967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.180936 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6lw\" (UniqueName: \"kubernetes.io/projected/6d801efa-5982-4805-a41d-cf64ed79f1e3-kube-api-access-9g6lw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:16.180967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.180966 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d801efa-5982-4805-a41d-cf64ed79f1e3-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:16.795473 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.795446 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" Nov 02 19:43:16.795636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.795481 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6" event={"ID":"6d801efa-5982-4805-a41d-cf64ed79f1e3","Type":"ContainerDied","Data":"08af4f02b3a4a286d284bb01ad6c8a4f7a5bf224aa6f84574e30dd4695d3ad3a"} Nov 02 19:43:16.795636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:16.795539 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08af4f02b3a4a286d284bb01ad6c8a4f7a5bf224aa6f84574e30dd4695d3ad3a" Nov 02 19:43:17.705505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.705472 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f"] Nov 02 19:43:17.705984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.705970 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6d801efa-5982-4805-a41d-cf64ed79f1e3" containerName="revoke-keys" Nov 02 19:43:17.706028 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.705985 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d801efa-5982-4805-a41d-cf64ed79f1e3" containerName="revoke-keys" Nov 02 19:43:17.706076 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.706067 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d801efa-5982-4805-a41d-cf64ed79f1e3" containerName="revoke-keys" Nov 02 19:43:17.709646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.709626 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.786126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.786096 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8795bf-openshift-default-dockercfg-s7q56\"" Nov 02 19:43:17.797982 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.797950 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798009 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798039 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798274 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798248 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798394 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798376 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n242t\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-kube-api-access-n242t\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798470 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798453 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12faba46-6929-4454-9353-d310edbf42a9-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798544 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798593 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798568 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12faba46-6929-4454-9353-d310edbf42a9-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.798626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.798591 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.806088 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.806059 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f"] Nov 02 19:43:17.899882 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.899847 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12faba46-6929-4454-9353-d310edbf42a9-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.899894 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.899915 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12faba46-6929-4454-9353-d310edbf42a9-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.899939 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.899985 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900066 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900012 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900071 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900194 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900246 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n242t\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-kube-api-access-n242t\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900565 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900478 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900565 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900526 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900648 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.900762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.900717 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12faba46-6929-4454-9353-d310edbf42a9-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.901136 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.901110 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.902407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.902387 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.902674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.902656 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12faba46-6929-4454-9353-d310edbf42a9-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.953504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.953468 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:17.953663 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:17.953579 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n242t\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-kube-api-access-n242t\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:18.023962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.023931 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:18.442939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.442910 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f"] Nov 02 19:43:18.444524 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:18.444491 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12faba46_6929_4454_9353_d310edbf42a9.slice/crio-11adebac78f9b66610deaef25b5141039d07b783a96df71c51f8a778e635ec9f WatchSource:0}: Error finding container 11adebac78f9b66610deaef25b5141039d07b783a96df71c51f8a778e635ec9f: Status 404 returned error can't find the container with id 11adebac78f9b66610deaef25b5141039d07b783a96df71c51f8a778e635ec9f Nov 02 19:43:18.447368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.447288 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:43:18.447448 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.447420 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:43:18.447508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.447492 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:43:18.810957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.810920 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" event={"ID":"12faba46-6929-4454-9353-d310edbf42a9","Type":"ContainerStarted","Data":"0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf"} Nov 02 19:43:18.811446 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.810987 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" event={"ID":"12faba46-6929-4454-9353-d310edbf42a9","Type":"ContainerStarted","Data":"11adebac78f9b66610deaef25b5141039d07b783a96df71c51f8a778e635ec9f"} Nov 02 19:43:18.920154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:18.920106 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podStartSLOduration=1.920093348 podStartE2EDuration="1.920093348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:18.919170546 +0000 UTC m=+904.867826385" watchObservedRunningTime="2026-11-02 19:43:18.920093348 +0000 UTC m=+904.868749187" Nov 02 19:43:19.025036 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:19.024996 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:19.026739 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:19.026711 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 19:43:19.026862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:19.026784 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 19:43:20.024638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.024602 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 19:43:20.025145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.024657 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 19:43:20.468049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.467953 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q"] Nov 02 19:43:20.472502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.472479 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.510930 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.510902 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q"] Nov 02 19:43:20.521422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.521393 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:20.521557 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.521477 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:20.560603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.560553 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:20.560794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.560686 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:20.630978 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.630943 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.631146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.630994 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.631146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.631020 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrmt\" (UniqueName: \"kubernetes.io/projected/fbe3c9bb-e604-4042-aa48-752beaeda872-kube-api-access-5lrmt\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.732468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.732435 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.732692 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.732491 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.732692 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.732528 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrmt\" (UniqueName: \"kubernetes.io/projected/fbe3c9bb-e604-4042-aa48-752beaeda872-kube-api-access-5lrmt\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.733269 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.733243 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.734906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.734884 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.785367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.785332 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrmt\" (UniqueName: \"kubernetes.io/projected/fbe3c9bb-e604-4042-aa48-752beaeda872-kube-api-access-5lrmt\") pod \"maas-api-revoke-sub-871dd1ce-c9b8q\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:20.787158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:20.787134 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:21.024816 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:21.024725 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 19:43:21.024816 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:21.024785 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 19:43:21.199091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:21.199065 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q"] Nov 02 19:43:21.200947 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:21.200893 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe3c9bb_e604_4042_aa48_752beaeda872.slice/crio-58b07a23fb8225140caa99f6df3a9ba3fd3a6f862a343b5d8ccf11d36e566bb7 WatchSource:0}: Error finding container 58b07a23fb8225140caa99f6df3a9ba3fd3a6f862a343b5d8ccf11d36e566bb7: Status 404 returned error can't find the container with id 58b07a23fb8225140caa99f6df3a9ba3fd3a6f862a343b5d8ccf11d36e566bb7 Nov 02 19:43:21.837324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:21.837221 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" event={"ID":"fbe3c9bb-e604-4042-aa48-752beaeda872","Type":"ContainerStarted","Data":"b1047fd3c52440ea0cff21b15e3590f83b46acf44bc1672563c71a92de2dbb99"} Nov 02 19:43:21.837324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:21.837284 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" event={"ID":"fbe3c9bb-e604-4042-aa48-752beaeda872","Type":"ContainerStarted","Data":"58b07a23fb8225140caa99f6df3a9ba3fd3a6f862a343b5d8ccf11d36e566bb7"} Nov 02 19:43:22.024324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:22.024268 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 19:43:22.024492 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:22.024347 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 19:43:22.868693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:22.868658 4966 generic.go:356] "Generic (PLEG): container finished" podID="fbe3c9bb-e604-4042-aa48-752beaeda872" containerID="b1047fd3c52440ea0cff21b15e3590f83b46acf44bc1672563c71a92de2dbb99" exitCode=0 Nov 02 19:43:22.869156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:22.868739 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" event={"ID":"fbe3c9bb-e604-4042-aa48-752beaeda872","Type":"ContainerDied","Data":"b1047fd3c52440ea0cff21b15e3590f83b46acf44bc1672563c71a92de2dbb99"} Nov 02 19:43:23.024477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.024439 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 19:43:23.024660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.024500 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 19:43:23.299022 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.298991 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k"] Nov 02 19:43:23.303398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.303382 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.368349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.368081 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k"] Nov 02 19:43:23.392962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.392919 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.393142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.393079 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndv66\" (UniqueName: \"kubernetes.io/projected/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-kube-api-access-ndv66\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.393142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.393113 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.494552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.494509 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndv66\" (UniqueName: \"kubernetes.io/projected/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-kube-api-access-ndv66\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.494757 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.494573 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.494757 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.494623 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.495767 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.495734 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.497326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.497282 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.554366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.551866 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndv66\" (UniqueName: \"kubernetes.io/projected/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-kube-api-access-ndv66\") pod \"maas-api-revoke-sub-528c4cd5-97s5k\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:23.617858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:23.617819 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:24.024598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.024571 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 19:43:24.024942 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.024621 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 19:43:24.050542 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.048279 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k"] Nov 02 19:43:24.071908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.071584 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:24.103640 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.103610 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe3c9bb-e604-4042-aa48-752beaeda872-kube-api-access-5lrmt\" (UniqueName: \"kubernetes.io/projected/fbe3c9bb-e604-4042-aa48-752beaeda872-kube-api-access-5lrmt\") pod \"fbe3c9bb-e604-4042-aa48-752beaeda872\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " Nov 02 19:43:24.103806 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.103660 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-xks-ca\") pod \"fbe3c9bb-e604-4042-aa48-752beaeda872\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " Nov 02 19:43:24.103806 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.103694 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-openshift-ca\") pod \"fbe3c9bb-e604-4042-aa48-752beaeda872\" (UID: \"fbe3c9bb-e604-4042-aa48-752beaeda872\") " Nov 02 19:43:24.104133 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.104108 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-openshift-ca" pod "fbe3c9bb-e604-4042-aa48-752beaeda872" (UID: "fbe3c9bb-e604-4042-aa48-752beaeda872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:24.106241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.106201 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-xks-ca" pod "fbe3c9bb-e604-4042-aa48-752beaeda872" (UID: "fbe3c9bb-e604-4042-aa48-752beaeda872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:24.106366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.106286 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe3c9bb-e604-4042-aa48-752beaeda872-kube-api-access-5lrmt" pod "fbe3c9bb-e604-4042-aa48-752beaeda872" (UID: "fbe3c9bb-e604-4042-aa48-752beaeda872"). InnerVolumeSpecName "kube-api-access-5lrmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:24.206014 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.205977 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrmt\" (UniqueName: \"kubernetes.io/projected/fbe3c9bb-e604-4042-aa48-752beaeda872-kube-api-access-5lrmt\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:24.206452 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.206430 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:24.206548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.206462 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe3c9bb-e604-4042-aa48-752beaeda872-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:24.884168 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.884088 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" Nov 02 19:43:24.884168 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.884126 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q" event={"ID":"fbe3c9bb-e604-4042-aa48-752beaeda872","Type":"ContainerDied","Data":"58b07a23fb8225140caa99f6df3a9ba3fd3a6f862a343b5d8ccf11d36e566bb7"} Nov 02 19:43:24.884397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.884178 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b07a23fb8225140caa99f6df3a9ba3fd3a6f862a343b5d8ccf11d36e566bb7" Nov 02 19:43:24.886432 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.886405 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" event={"ID":"3bd0582e-c421-4fa1-8d7b-df8bd79961e1","Type":"ContainerStarted","Data":"2ed7e0cff74ec632c84cce65cbcfa053afb42df15e72cf849ea51c39ec2abcc0"} Nov 02 19:43:24.886632 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.886454 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" event={"ID":"3bd0582e-c421-4fa1-8d7b-df8bd79961e1","Type":"ContainerStarted","Data":"95e56c52ee3455e55de7ef6cf44cfa259f3f0ec2ab0fcae40d2e8f5593bebe63"} Nov 02 19:43:24.995796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:24.995736 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" podStartSLOduration=1.995721435 podStartE2EDuration="1.995721435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:24.993584555 +0000 UTC m=+910.942240395" watchObservedRunningTime="2026-11-02 19:43:24.995721435 +0000 UTC m=+910.944377266" Nov 02 19:43:25.209466 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:25.209371 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:25.210350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:25.210292 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:25.211237 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:25.211212 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:43:25.894363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:25.894323 4966 generic.go:356] "Generic (PLEG): container finished" podID="3bd0582e-c421-4fa1-8d7b-df8bd79961e1" containerID="2ed7e0cff74ec632c84cce65cbcfa053afb42df15e72cf849ea51c39ec2abcc0" exitCode=0 Nov 02 19:43:25.894574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:25.894417 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" event={"ID":"3bd0582e-c421-4fa1-8d7b-df8bd79961e1","Type":"ContainerDied","Data":"2ed7e0cff74ec632c84cce65cbcfa053afb42df15e72cf849ea51c39ec2abcc0"} Nov 02 19:43:27.070992 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.070962 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:27.144407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.144374 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-xks-ca\") pod \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " Nov 02 19:43:27.144589 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.144430 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-kube-api-access-ndv66\" (UniqueName: \"kubernetes.io/projected/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-kube-api-access-ndv66\") pod \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " Nov 02 19:43:27.144589 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.144482 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-openshift-ca\") pod \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\" (UID: \"3bd0582e-c421-4fa1-8d7b-df8bd79961e1\") " Nov 02 19:43:27.144903 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.144879 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-openshift-ca" pod "3bd0582e-c421-4fa1-8d7b-df8bd79961e1" (UID: "3bd0582e-c421-4fa1-8d7b-df8bd79961e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:27.146566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.146540 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-kube-api-access-ndv66" pod "3bd0582e-c421-4fa1-8d7b-df8bd79961e1" (UID: "3bd0582e-c421-4fa1-8d7b-df8bd79961e1"). InnerVolumeSpecName "kube-api-access-ndv66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:27.146670 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.146582 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-xks-ca" pod "3bd0582e-c421-4fa1-8d7b-df8bd79961e1" (UID: "3bd0582e-c421-4fa1-8d7b-df8bd79961e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:27.245857 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.245825 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:27.245857 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.245853 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndv66\" (UniqueName: \"kubernetes.io/projected/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-kube-api-access-ndv66\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:27.245857 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.245863 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0582e-c421-4fa1-8d7b-df8bd79961e1-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:27.908509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.908476 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" Nov 02 19:43:27.908678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.908510 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k" event={"ID":"3bd0582e-c421-4fa1-8d7b-df8bd79961e1","Type":"ContainerDied","Data":"95e56c52ee3455e55de7ef6cf44cfa259f3f0ec2ab0fcae40d2e8f5593bebe63"} Nov 02 19:43:27.908678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:27.908573 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95e56c52ee3455e55de7ef6cf44cfa259f3f0ec2ab0fcae40d2e8f5593bebe63" Nov 02 19:43:28.411096 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.411055 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq"] Nov 02 19:43:28.412406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.412378 4966 cpu_manager.go:414] "containerMap: removing container" podUID="fbe3c9bb-e604-4042-aa48-752beaeda872" containerName="revoke-keys" Nov 02 19:43:28.412597 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.412575 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe3c9bb-e604-4042-aa48-752beaeda872" containerName="revoke-keys" Nov 02 19:43:28.412722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.412696 4966 cpu_manager.go:414] "containerMap: removing container" podUID="3bd0582e-c421-4fa1-8d7b-df8bd79961e1" containerName="revoke-keys" Nov 02 19:43:28.412722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.412706 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd0582e-c421-4fa1-8d7b-df8bd79961e1" containerName="revoke-keys" Nov 02 19:43:28.413024 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.413001 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd0582e-c421-4fa1-8d7b-df8bd79961e1" containerName="revoke-keys" Nov 02 19:43:28.413116 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.413040 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe3c9bb-e604-4042-aa48-752beaeda872" containerName="revoke-keys" Nov 02 19:43:28.417925 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.417903 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.458487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.458290 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001f1027-384c-4afe-b6c7-119968977c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.458487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.458390 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001f1027-384c-4afe-b6c7-119968977c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.458678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.458492 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44rj\" (UniqueName: \"kubernetes.io/projected/001f1027-384c-4afe-b6c7-119968977c98-kube-api-access-d44rj\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.467515 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.467472 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq"] Nov 02 19:43:28.504053 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.504021 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:28.508700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.508678 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:28.508811 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.508783 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:28.508811 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.508782 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:28.559504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.559461 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001f1027-384c-4afe-b6c7-119968977c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.559689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.559532 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44rj\" (UniqueName: \"kubernetes.io/projected/001f1027-384c-4afe-b6c7-119968977c98-kube-api-access-d44rj\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.559689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.559612 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001f1027-384c-4afe-b6c7-119968977c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.560272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.560246 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001f1027-384c-4afe-b6c7-119968977c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.562048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.562022 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001f1027-384c-4afe-b6c7-119968977c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.612770 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.612733 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44rj\" (UniqueName: \"kubernetes.io/projected/001f1027-384c-4afe-b6c7-119968977c98-kube-api-access-d44rj\") pod \"maas-api-revoke-sub-8c045110-zjrtq\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.736350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.736288 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:28.931796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.931764 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp"] Nov 02 19:43:28.936939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.936915 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:28.964257 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.964223 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:28.964418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.964321 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:28.964470 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.964415 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lgt\" (UniqueName: \"kubernetes.io/projected/dd4f2c2e-e36d-449f-99dd-029fd965d56a-kube-api-access-l4lgt\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:28.990534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:28.990444 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp"] Nov 02 19:43:29.065686 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.065649 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:29.065903 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.065698 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lgt\" (UniqueName: \"kubernetes.io/projected/dd4f2c2e-e36d-449f-99dd-029fd965d56a-kube-api-access-l4lgt\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:29.065903 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.065792 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:29.066522 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.066500 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:29.068089 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.068062 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:29.138338 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.138289 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lgt\" (UniqueName: \"kubernetes.io/projected/dd4f2c2e-e36d-449f-99dd-029fd965d56a-kube-api-access-l4lgt\") pod \"maas-api-revoke-sub-30926ff4-qjzwp\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:29.164975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.164942 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq"] Nov 02 19:43:29.165793 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:29.165768 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001f1027_384c_4afe_b6c7_119968977c98.slice/crio-f9cf2de235a59538923bcfb872128414f7c7d0d33d3edc0a2047ff4fe37f85c9 WatchSource:0}: Error finding container f9cf2de235a59538923bcfb872128414f7c7d0d33d3edc0a2047ff4fe37f85c9: Status 404 returned error can't find the container with id f9cf2de235a59538923bcfb872128414f7c7d0d33d3edc0a2047ff4fe37f85c9 Nov 02 19:43:29.251808 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.251719 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:29.622382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.622353 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp"] Nov 02 19:43:29.623210 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:29.623184 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4f2c2e_e36d_449f_99dd_029fd965d56a.slice/crio-0d408d1afada073e3bdb50f6ff8de7e515e2ab4272216ecc970acfc67fca3c1e WatchSource:0}: Error finding container 0d408d1afada073e3bdb50f6ff8de7e515e2ab4272216ecc970acfc67fca3c1e: Status 404 returned error can't find the container with id 0d408d1afada073e3bdb50f6ff8de7e515e2ab4272216ecc970acfc67fca3c1e Nov 02 19:43:29.923960 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.923846 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" event={"ID":"001f1027-384c-4afe-b6c7-119968977c98","Type":"ContainerStarted","Data":"bb2441a95e4c30a71bcda836510dbd577e1c912303a10b0b27c0b85ac8e1cae4"} Nov 02 19:43:29.923960 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.923922 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" event={"ID":"001f1027-384c-4afe-b6c7-119968977c98","Type":"ContainerStarted","Data":"f9cf2de235a59538923bcfb872128414f7c7d0d33d3edc0a2047ff4fe37f85c9"} Nov 02 19:43:29.925373 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:29.925346 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" event={"ID":"dd4f2c2e-e36d-449f-99dd-029fd965d56a","Type":"ContainerStarted","Data":"0d408d1afada073e3bdb50f6ff8de7e515e2ab4272216ecc970acfc67fca3c1e"} Nov 02 19:43:30.031675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:30.031620 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" podStartSLOduration=2.031606409 podStartE2EDuration="2.031606409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:30.029531248 +0000 UTC m=+915.978187088" watchObservedRunningTime="2026-11-02 19:43:30.031606409 +0000 UTC m=+915.980262283" Nov 02 19:43:30.933438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:30.933352 4966 generic.go:356] "Generic (PLEG): container finished" podID="001f1027-384c-4afe-b6c7-119968977c98" containerID="bb2441a95e4c30a71bcda836510dbd577e1c912303a10b0b27c0b85ac8e1cae4" exitCode=0 Nov 02 19:43:30.933874 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:30.933430 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" event={"ID":"001f1027-384c-4afe-b6c7-119968977c98","Type":"ContainerDied","Data":"bb2441a95e4c30a71bcda836510dbd577e1c912303a10b0b27c0b85ac8e1cae4"} Nov 02 19:43:30.935063 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:30.935028 4966 generic.go:356] "Generic (PLEG): container finished" podID="dd4f2c2e-e36d-449f-99dd-029fd965d56a" containerID="e0ccc39be1e9433ff1c168cd1d831a48a538631f2a2dd8146b3f9a728b2a7094" exitCode=0 Nov 02 19:43:30.935063 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:30.935063 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" event={"ID":"dd4f2c2e-e36d-449f-99dd-029fd965d56a","Type":"ContainerDied","Data":"e0ccc39be1e9433ff1c168cd1d831a48a538631f2a2dd8146b3f9a728b2a7094"} Nov 02 19:43:32.121321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.121287 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:32.124540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.124524 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:32.198604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.198541 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-xks-ca\") pod \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " Nov 02 19:43:32.198731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.198645 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001f1027-384c-4afe-b6c7-119968977c98-kube-api-access-d44rj\" (UniqueName: \"kubernetes.io/projected/001f1027-384c-4afe-b6c7-119968977c98-kube-api-access-d44rj\") pod \"001f1027-384c-4afe-b6c7-119968977c98\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " Nov 02 19:43:32.198731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.198677 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-openshift-ca\") pod \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " Nov 02 19:43:32.198802 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.198737 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001f1027-384c-4afe-b6c7-119968977c98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001f1027-384c-4afe-b6c7-119968977c98-maas-api-xks-ca\") pod \"001f1027-384c-4afe-b6c7-119968977c98\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " Nov 02 19:43:32.198802 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.198775 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001f1027-384c-4afe-b6c7-119968977c98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001f1027-384c-4afe-b6c7-119968977c98-maas-api-openshift-ca\") pod \"001f1027-384c-4afe-b6c7-119968977c98\" (UID: \"001f1027-384c-4afe-b6c7-119968977c98\") " Nov 02 19:43:32.198901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.198826 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4f2c2e-e36d-449f-99dd-029fd965d56a-kube-api-access-l4lgt\" (UniqueName: \"kubernetes.io/projected/dd4f2c2e-e36d-449f-99dd-029fd965d56a-kube-api-access-l4lgt\") pod \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\" (UID: \"dd4f2c2e-e36d-449f-99dd-029fd965d56a\") " Nov 02 19:43:32.199145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.199117 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-openshift-ca" pod "dd4f2c2e-e36d-449f-99dd-029fd965d56a" (UID: "dd4f2c2e-e36d-449f-99dd-029fd965d56a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:32.199261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.199159 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001f1027-384c-4afe-b6c7-119968977c98-maas-api-openshift-ca" pod "001f1027-384c-4afe-b6c7-119968977c98" (UID: "001f1027-384c-4afe-b6c7-119968977c98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:32.199390 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.199362 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:32.199499 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.199392 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001f1027-384c-4afe-b6c7-119968977c98-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:32.200766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.200739 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-xks-ca" pod "dd4f2c2e-e36d-449f-99dd-029fd965d56a" (UID: "dd4f2c2e-e36d-449f-99dd-029fd965d56a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:32.200850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.200777 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001f1027-384c-4afe-b6c7-119968977c98-maas-api-xks-ca" pod "001f1027-384c-4afe-b6c7-119968977c98" (UID: "001f1027-384c-4afe-b6c7-119968977c98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:32.200898 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.200853 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001f1027-384c-4afe-b6c7-119968977c98-kube-api-access-d44rj" pod "001f1027-384c-4afe-b6c7-119968977c98" (UID: "001f1027-384c-4afe-b6c7-119968977c98"). InnerVolumeSpecName "kube-api-access-d44rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:32.200951 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.200931 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4f2c2e-e36d-449f-99dd-029fd965d56a-kube-api-access-l4lgt" pod "dd4f2c2e-e36d-449f-99dd-029fd965d56a" (UID: "dd4f2c2e-e36d-449f-99dd-029fd965d56a"). InnerVolumeSpecName "kube-api-access-l4lgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:32.301313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.301258 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4lgt\" (UniqueName: \"kubernetes.io/projected/dd4f2c2e-e36d-449f-99dd-029fd965d56a-kube-api-access-l4lgt\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:32.301313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.301292 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd4f2c2e-e36d-449f-99dd-029fd965d56a-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:32.301554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.301331 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44rj\" (UniqueName: \"kubernetes.io/projected/001f1027-384c-4afe-b6c7-119968977c98-kube-api-access-d44rj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:32.301554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.301348 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001f1027-384c-4afe-b6c7-119968977c98-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:32.953327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.953281 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" Nov 02 19:43:32.953524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.953322 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp" event={"ID":"dd4f2c2e-e36d-449f-99dd-029fd965d56a","Type":"ContainerDied","Data":"0d408d1afada073e3bdb50f6ff8de7e515e2ab4272216ecc970acfc67fca3c1e"} Nov 02 19:43:32.953524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.953426 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d408d1afada073e3bdb50f6ff8de7e515e2ab4272216ecc970acfc67fca3c1e" Nov 02 19:43:32.954924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.954907 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" Nov 02 19:43:32.955048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.954906 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq" event={"ID":"001f1027-384c-4afe-b6c7-119968977c98","Type":"ContainerDied","Data":"f9cf2de235a59538923bcfb872128414f7c7d0d33d3edc0a2047ff4fe37f85c9"} Nov 02 19:43:32.955048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:32.955014 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9cf2de235a59538923bcfb872128414f7c7d0d33d3edc0a2047ff4fe37f85c9" Nov 02 19:43:36.501937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.501901 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc"] Nov 02 19:43:36.502444 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.502415 4966 cpu_manager.go:414] "containerMap: removing container" podUID="001f1027-384c-4afe-b6c7-119968977c98" containerName="revoke-keys" Nov 02 19:43:36.502444 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.502427 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001f1027-384c-4afe-b6c7-119968977c98" containerName="revoke-keys" Nov 02 19:43:36.502515 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.502454 4966 cpu_manager.go:414] "containerMap: removing container" podUID="dd4f2c2e-e36d-449f-99dd-029fd965d56a" containerName="revoke-keys" Nov 02 19:43:36.502515 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.502459 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4f2c2e-e36d-449f-99dd-029fd965d56a" containerName="revoke-keys" Nov 02 19:43:36.502573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.502535 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="001f1027-384c-4afe-b6c7-119968977c98" containerName="revoke-keys" Nov 02 19:43:36.502573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.502548 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4f2c2e-e36d-449f-99dd-029fd965d56a" containerName="revoke-keys" Nov 02 19:43:36.506350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.506333 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.561346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.561310 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc"] Nov 02 19:43:36.602680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.602640 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:36.602840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.602707 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:36.602840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.602729 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:36.602840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.602814 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:36.650267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.650228 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j95tz\" (UniqueName: \"kubernetes.io/projected/466eced8-8d4f-4691-9b43-e96e7af148e5-kube-api-access-j95tz\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.650442 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.650411 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.650522 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.650445 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.751763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.751721 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j95tz\" (UniqueName: \"kubernetes.io/projected/466eced8-8d4f-4691-9b43-e96e7af148e5-kube-api-access-j95tz\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.752013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.751833 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.752013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.751856 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.752622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.752598 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.754320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.754268 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.804651 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.804621 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j95tz\" (UniqueName: \"kubernetes.io/projected/466eced8-8d4f-4691-9b43-e96e7af148e5-kube-api-access-j95tz\") pod \"maas-api-revoke-sub-d07dbafe-zcnbc\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:36.820463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:36.820429 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:37.196474 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:37.195231 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc"] Nov 02 19:43:37.994917 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:37.994882 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" event={"ID":"466eced8-8d4f-4691-9b43-e96e7af148e5","Type":"ContainerStarted","Data":"1304b8bee89b2d906966e3cd249aeea464881b80e7c0c5781025f6661df7a70a"} Nov 02 19:43:37.995350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:37.994939 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" event={"ID":"466eced8-8d4f-4691-9b43-e96e7af148e5","Type":"ContainerStarted","Data":"1b27d3c385fcd415cd2b1943367adf3aaee98293daf5bf9244e5134a7f67b823"} Nov 02 19:43:38.104539 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:38.104485 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" podStartSLOduration=2.104470733 podStartE2EDuration="2.104470733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:38.100580801 +0000 UTC m=+924.049236661" watchObservedRunningTime="2026-11-02 19:43:38.104470733 +0000 UTC m=+924.053126567" Nov 02 19:43:39.003274 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:39.003241 4966 generic.go:356] "Generic (PLEG): container finished" podID="466eced8-8d4f-4691-9b43-e96e7af148e5" containerID="1304b8bee89b2d906966e3cd249aeea464881b80e7c0c5781025f6661df7a70a" exitCode=0 Nov 02 19:43:39.003673 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:39.003332 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" event={"ID":"466eced8-8d4f-4691-9b43-e96e7af148e5","Type":"ContainerDied","Data":"1304b8bee89b2d906966e3cd249aeea464881b80e7c0c5781025f6661df7a70a"} Nov 02 19:43:40.131746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.131708 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98"] Nov 02 19:43:40.137123 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.137100 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.189831 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.189794 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98"] Nov 02 19:43:40.200270 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.200246 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:40.292365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.292257 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-xks-ca\") pod \"466eced8-8d4f-4691-9b43-e96e7af148e5\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " Nov 02 19:43:40.292548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.292368 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466eced8-8d4f-4691-9b43-e96e7af148e5-kube-api-access-j95tz\" (UniqueName: \"kubernetes.io/projected/466eced8-8d4f-4691-9b43-e96e7af148e5-kube-api-access-j95tz\") pod \"466eced8-8d4f-4691-9b43-e96e7af148e5\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " Nov 02 19:43:40.292548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.292409 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-openshift-ca\") pod \"466eced8-8d4f-4691-9b43-e96e7af148e5\" (UID: \"466eced8-8d4f-4691-9b43-e96e7af148e5\") " Nov 02 19:43:40.292721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.292701 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skhs\" (UniqueName: \"kubernetes.io/projected/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-kube-api-access-5skhs\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.292850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.292824 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-openshift-ca" pod "466eced8-8d4f-4691-9b43-e96e7af148e5" (UID: "466eced8-8d4f-4691-9b43-e96e7af148e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:40.293401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.292934 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.293401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.293085 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.293401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.293171 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:40.294611 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.294578 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466eced8-8d4f-4691-9b43-e96e7af148e5-kube-api-access-j95tz" pod "466eced8-8d4f-4691-9b43-e96e7af148e5" (UID: "466eced8-8d4f-4691-9b43-e96e7af148e5"). InnerVolumeSpecName "kube-api-access-j95tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:40.294713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.294632 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-xks-ca" pod "466eced8-8d4f-4691-9b43-e96e7af148e5" (UID: "466eced8-8d4f-4691-9b43-e96e7af148e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:40.394213 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.394170 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skhs\" (UniqueName: \"kubernetes.io/projected/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-kube-api-access-5skhs\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.394402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.394277 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.394402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.394357 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.394509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.394409 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/466eced8-8d4f-4691-9b43-e96e7af148e5-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:40.394509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.394420 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j95tz\" (UniqueName: \"kubernetes.io/projected/466eced8-8d4f-4691-9b43-e96e7af148e5-kube-api-access-j95tz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:40.394962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.394941 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.396822 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.396802 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.447317 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.447274 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skhs\" (UniqueName: \"kubernetes.io/projected/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-kube-api-access-5skhs\") pod \"maas-api-revoke-sub-e2822e1b-kzl98\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.498387 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.498350 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:40.866029 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:40.865999 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98"] Nov 02 19:43:40.866522 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:40.866497 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c48a06_a93d_48a7_9b8a_c80e59ba650b.slice/crio-01557b80b1a1b71397c185d3e720b98b1597e552791cc20f8735c8bb0fe3b54f WatchSource:0}: Error finding container 01557b80b1a1b71397c185d3e720b98b1597e552791cc20f8735c8bb0fe3b54f: Status 404 returned error can't find the container with id 01557b80b1a1b71397c185d3e720b98b1597e552791cc20f8735c8bb0fe3b54f Nov 02 19:43:41.018991 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:41.018951 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" event={"ID":"f9c48a06-a93d-48a7-9b8a-c80e59ba650b","Type":"ContainerStarted","Data":"01557b80b1a1b71397c185d3e720b98b1597e552791cc20f8735c8bb0fe3b54f"} Nov 02 19:43:41.020382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:41.020357 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" event={"ID":"466eced8-8d4f-4691-9b43-e96e7af148e5","Type":"ContainerDied","Data":"1b27d3c385fcd415cd2b1943367adf3aaee98293daf5bf9244e5134a7f67b823"} Nov 02 19:43:41.020509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:41.020405 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b27d3c385fcd415cd2b1943367adf3aaee98293daf5bf9244e5134a7f67b823" Nov 02 19:43:41.020509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:41.020361 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc" Nov 02 19:43:42.028768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:42.028685 4966 generic.go:356] "Generic (PLEG): container finished" podID="f9c48a06-a93d-48a7-9b8a-c80e59ba650b" containerID="d450354bae64e217f54b2ac9a188700dd38ebcd79a198852ef3430f463b92ac2" exitCode=0 Nov 02 19:43:42.029159 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:42.028770 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" event={"ID":"f9c48a06-a93d-48a7-9b8a-c80e59ba650b","Type":"ContainerDied","Data":"d450354bae64e217f54b2ac9a188700dd38ebcd79a198852ef3430f463b92ac2"} Nov 02 19:43:43.191699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.191671 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:43.327212 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.327143 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-kube-api-access-5skhs\" (UniqueName: \"kubernetes.io/projected/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-kube-api-access-5skhs\") pod \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " Nov 02 19:43:43.327381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.327253 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-openshift-ca\") pod \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " Nov 02 19:43:43.327426 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.327387 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-xks-ca\") pod \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\" (UID: \"f9c48a06-a93d-48a7-9b8a-c80e59ba650b\") " Nov 02 19:43:43.327659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.327623 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-openshift-ca" pod "f9c48a06-a93d-48a7-9b8a-c80e59ba650b" (UID: "f9c48a06-a93d-48a7-9b8a-c80e59ba650b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:43.327841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.327818 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:43.329558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.329535 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-xks-ca" pod "f9c48a06-a93d-48a7-9b8a-c80e59ba650b" (UID: "f9c48a06-a93d-48a7-9b8a-c80e59ba650b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:43.329664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.329636 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-kube-api-access-5skhs" pod "f9c48a06-a93d-48a7-9b8a-c80e59ba650b" (UID: "f9c48a06-a93d-48a7-9b8a-c80e59ba650b"). InnerVolumeSpecName "kube-api-access-5skhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:43.428782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.428749 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5skhs\" (UniqueName: \"kubernetes.io/projected/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-kube-api-access-5skhs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:43.428782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.428777 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c48a06-a93d-48a7-9b8a-c80e59ba650b-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:43.966911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.966876 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh"] Nov 02 19:43:43.967424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.967407 4966 cpu_manager.go:414] "containerMap: removing container" podUID="466eced8-8d4f-4691-9b43-e96e7af148e5" containerName="revoke-keys" Nov 02 19:43:43.967424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.967423 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466eced8-8d4f-4691-9b43-e96e7af148e5" containerName="revoke-keys" Nov 02 19:43:43.967544 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.967436 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f9c48a06-a93d-48a7-9b8a-c80e59ba650b" containerName="revoke-keys" Nov 02 19:43:43.967544 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.967441 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c48a06-a93d-48a7-9b8a-c80e59ba650b" containerName="revoke-keys" Nov 02 19:43:43.967544 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.967518 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c48a06-a93d-48a7-9b8a-c80e59ba650b" containerName="revoke-keys" Nov 02 19:43:43.967544 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.967528 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="466eced8-8d4f-4691-9b43-e96e7af148e5" containerName="revoke-keys" Nov 02 19:43:43.970979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:43.970958 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.025423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.025390 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh"] Nov 02 19:43:44.046793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.046755 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" event={"ID":"f9c48a06-a93d-48a7-9b8a-c80e59ba650b","Type":"ContainerDied","Data":"01557b80b1a1b71397c185d3e720b98b1597e552791cc20f8735c8bb0fe3b54f"} Nov 02 19:43:44.047006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.046823 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01557b80b1a1b71397c185d3e720b98b1597e552791cc20f8735c8bb0fe3b54f" Nov 02 19:43:44.047006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.046898 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98" Nov 02 19:43:44.137615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.137579 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.137761 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.137636 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqt5p\" (UniqueName: \"kubernetes.io/projected/ebd50657-fcc9-44f0-b811-72a952a22fb8-kube-api-access-sqt5p\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.137814 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.137787 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.239069 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.239029 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.239498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.239096 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqt5p\" (UniqueName: \"kubernetes.io/projected/ebd50657-fcc9-44f0-b811-72a952a22fb8-kube-api-access-sqt5p\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.239498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.239212 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.239877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.239849 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.241775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.241747 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.291937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.291906 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqt5p\" (UniqueName: \"kubernetes.io/projected/ebd50657-fcc9-44f0-b811-72a952a22fb8-kube-api-access-sqt5p\") pod \"maas-api-revoke-sub-79202cf8-tvqvh\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.584788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.584705 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:44.753429 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:43:44.753394 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:43:44.958088 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:44.958060 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd50657_fcc9_44f0_b811_72a952a22fb8.slice/crio-c409d5582fb41d5ca5da5b39237e2b926382a67df2101614cedb97083e60453f WatchSource:0}: Error finding container c409d5582fb41d5ca5da5b39237e2b926382a67df2101614cedb97083e60453f: Status 404 returned error can't find the container with id c409d5582fb41d5ca5da5b39237e2b926382a67df2101614cedb97083e60453f Nov 02 19:43:44.958806 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:44.958780 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh"] Nov 02 19:43:45.055643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.055608 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" event={"ID":"ebd50657-fcc9-44f0-b811-72a952a22fb8","Type":"ContainerStarted","Data":"c409d5582fb41d5ca5da5b39237e2b926382a67df2101614cedb97083e60453f"} Nov 02 19:43:45.689351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.689249 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt"] Nov 02 19:43:45.693326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.693288 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.748524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.748488 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt"] Nov 02 19:43:45.857992 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.857953 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv45x\" (UniqueName: \"kubernetes.io/projected/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-kube-api-access-nv45x\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.858170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.858014 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.858170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.858143 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.959459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.959356 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv45x\" (UniqueName: \"kubernetes.io/projected/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-kube-api-access-nv45x\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.959459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.959417 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.959459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.959456 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.960091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.960059 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:45.961934 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:45.961910 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:46.014558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:46.014527 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv45x\" (UniqueName: \"kubernetes.io/projected/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-kube-api-access-nv45x\") pod \"maas-api-revoke-sub-3744a939-x6snt\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:46.064341 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:46.064278 4966 generic.go:356] "Generic (PLEG): container finished" podID="ebd50657-fcc9-44f0-b811-72a952a22fb8" containerID="8f2dd28785cc6023a180ef7c6f86692fe4b4039f6cd1601a060570bff0ee480d" exitCode=0 Nov 02 19:43:46.064500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:46.064370 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" event={"ID":"ebd50657-fcc9-44f0-b811-72a952a22fb8","Type":"ContainerDied","Data":"8f2dd28785cc6023a180ef7c6f86692fe4b4039f6cd1601a060570bff0ee480d"} Nov 02 19:43:46.307468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:46.307434 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:46.688645 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:46.688617 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt"] Nov 02 19:43:46.689602 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:46.689564 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2da2b11_aa7f_45b5_b4dc_0ff279c1ba93.slice/crio-59c0317e93527aeae9e2be423880ae55f7fb1858c0d22ff184ef5afba22ea74a WatchSource:0}: Error finding container 59c0317e93527aeae9e2be423880ae55f7fb1858c0d22ff184ef5afba22ea74a: Status 404 returned error can't find the container with id 59c0317e93527aeae9e2be423880ae55f7fb1858c0d22ff184ef5afba22ea74a Nov 02 19:43:47.074100 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.073957 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" event={"ID":"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93","Type":"ContainerStarted","Data":"bf2954401a34244d943d6b4e7aa633b1cbf7310b2e09ca148a04de2a4333a97a"} Nov 02 19:43:47.074100 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.074034 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" event={"ID":"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93","Type":"ContainerStarted","Data":"59c0317e93527aeae9e2be423880ae55f7fb1858c0d22ff184ef5afba22ea74a"} Nov 02 19:43:47.183067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.183002 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" podStartSLOduration=2.182982558 podStartE2EDuration="2.182982558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:47.18217706 +0000 UTC m=+933.130832912" watchObservedRunningTime="2026-11-02 19:43:47.182982558 +0000 UTC m=+933.131638397" Nov 02 19:43:47.231340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.231292 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:47.376325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.376265 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd50657-fcc9-44f0-b811-72a952a22fb8-kube-api-access-sqt5p\" (UniqueName: \"kubernetes.io/projected/ebd50657-fcc9-44f0-b811-72a952a22fb8-kube-api-access-sqt5p\") pod \"ebd50657-fcc9-44f0-b811-72a952a22fb8\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " Nov 02 19:43:47.376516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.376345 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-openshift-ca\") pod \"ebd50657-fcc9-44f0-b811-72a952a22fb8\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " Nov 02 19:43:47.376516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.376407 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-xks-ca\") pod \"ebd50657-fcc9-44f0-b811-72a952a22fb8\" (UID: \"ebd50657-fcc9-44f0-b811-72a952a22fb8\") " Nov 02 19:43:47.377119 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.377079 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-openshift-ca" pod "ebd50657-fcc9-44f0-b811-72a952a22fb8" (UID: "ebd50657-fcc9-44f0-b811-72a952a22fb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:47.378860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.378801 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-xks-ca" pod "ebd50657-fcc9-44f0-b811-72a952a22fb8" (UID: "ebd50657-fcc9-44f0-b811-72a952a22fb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:47.379377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.379348 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd50657-fcc9-44f0-b811-72a952a22fb8-kube-api-access-sqt5p" pod "ebd50657-fcc9-44f0-b811-72a952a22fb8" (UID: "ebd50657-fcc9-44f0-b811-72a952a22fb8"). InnerVolumeSpecName "kube-api-access-sqt5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:47.478324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.478273 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqt5p\" (UniqueName: \"kubernetes.io/projected/ebd50657-fcc9-44f0-b811-72a952a22fb8-kube-api-access-sqt5p\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:47.478324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.478325 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:47.478536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:47.478342 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd50657-fcc9-44f0-b811-72a952a22fb8-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:48.082226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:48.082187 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" Nov 02 19:43:48.082226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:48.082197 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh" event={"ID":"ebd50657-fcc9-44f0-b811-72a952a22fb8","Type":"ContainerDied","Data":"c409d5582fb41d5ca5da5b39237e2b926382a67df2101614cedb97083e60453f"} Nov 02 19:43:48.082816 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:48.082254 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c409d5582fb41d5ca5da5b39237e2b926382a67df2101614cedb97083e60453f" Nov 02 19:43:48.084260 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:48.084223 4966 generic.go:356] "Generic (PLEG): container finished" podID="d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" containerID="bf2954401a34244d943d6b4e7aa633b1cbf7310b2e09ca148a04de2a4333a97a" exitCode=0 Nov 02 19:43:48.084413 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:48.084268 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" event={"ID":"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93","Type":"ContainerDied","Data":"bf2954401a34244d943d6b4e7aa633b1cbf7310b2e09ca148a04de2a4333a97a"} Nov 02 19:43:49.260940 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.260915 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:49.402740 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.402655 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-openshift-ca\") pod \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " Nov 02 19:43:49.402909 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.402859 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-xks-ca\") pod \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " Nov 02 19:43:49.402978 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.402944 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-kube-api-access-nv45x\" (UniqueName: \"kubernetes.io/projected/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-kube-api-access-nv45x\") pod \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\" (UID: \"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93\") " Nov 02 19:43:49.403064 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.403034 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-openshift-ca" pod "d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" (UID: "d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:49.403355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.403329 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:49.405097 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.405071 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-xks-ca" pod "d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" (UID: "d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:49.405569 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.405543 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-kube-api-access-nv45x" pod "d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" (UID: "d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93"). InnerVolumeSpecName "kube-api-access-nv45x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:49.504942 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.504905 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:49.504942 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.504939 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv45x\" (UniqueName: \"kubernetes.io/projected/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93-kube-api-access-nv45x\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:49.612309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.612264 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k"] Nov 02 19:43:49.612840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.612826 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" containerName="revoke-keys" Nov 02 19:43:49.612878 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.612842 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" containerName="revoke-keys" Nov 02 19:43:49.612878 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.612859 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ebd50657-fcc9-44f0-b811-72a952a22fb8" containerName="revoke-keys" Nov 02 19:43:49.612878 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.612864 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd50657-fcc9-44f0-b811-72a952a22fb8" containerName="revoke-keys" Nov 02 19:43:49.613023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.613014 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" containerName="revoke-keys" Nov 02 19:43:49.613054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.613025 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd50657-fcc9-44f0-b811-72a952a22fb8" containerName="revoke-keys" Nov 02 19:43:49.616655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.616635 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.670127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.668509 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"distinct-w6-e9c144-kserve-self-signed-certs\"" Nov 02 19:43:49.672773 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.672744 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k"] Nov 02 19:43:49.808092 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.808052 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-dshm\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.808261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.808200 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnnk\" (UniqueName: \"kubernetes.io/projected/772633d4-a7f3-47f3-a51e-fc351e9e366c-kube-api-access-dmnnk\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.808326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.808267 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772633d4-a7f3-47f3-a51e-fc351e9e366c-tls-certs\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.808416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.808395 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-tmp-dir\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.808500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.808454 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-home\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.808554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.808532 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-model-cache\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909278 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909223 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772633d4-a7f3-47f3-a51e-fc351e9e366c-tls-certs\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909332 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-tmp-dir\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909386 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-home\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909436 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-model-cache\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909469 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-dshm\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909554 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnnk\" (UniqueName: \"kubernetes.io/projected/772633d4-a7f3-47f3-a51e-fc351e9e366c-kube-api-access-dmnnk\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909865 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-tmp-dir\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.909981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.909897 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-model-cache\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.910236 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.910207 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-home\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.912355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.912310 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772633d4-a7f3-47f3-a51e-fc351e9e366c-tls-certs\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.912442 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.912404 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-dshm\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:49.964377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:49.963465 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnnk\" (UniqueName: \"kubernetes.io/projected/772633d4-a7f3-47f3-a51e-fc351e9e366c-kube-api-access-dmnnk\") pod \"distinct-w6-e9c144-kserve-564c795857-njq2k\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:50.100458 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.100431 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" Nov 02 19:43:50.100613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.100463 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt" event={"ID":"d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93","Type":"ContainerDied","Data":"59c0317e93527aeae9e2be423880ae55f7fb1858c0d22ff184ef5afba22ea74a"} Nov 02 19:43:50.100613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.100510 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59c0317e93527aeae9e2be423880ae55f7fb1858c0d22ff184ef5afba22ea74a" Nov 02 19:43:50.230494 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.230449 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:50.610564 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.608161 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k"] Nov 02 19:43:50.611769 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:50.611736 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod772633d4_a7f3_47f3_a51e_fc351e9e366c.slice/crio-99808b00d3c41c62a21bdf398c23234f384432851b561e4bb0ac68019103cf46 WatchSource:0}: Error finding container 99808b00d3c41c62a21bdf398c23234f384432851b561e4bb0ac68019103cf46: Status 404 returned error can't find the container with id 99808b00d3c41c62a21bdf398c23234f384432851b561e4bb0ac68019103cf46 Nov 02 19:43:50.864714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.864633 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6"] Nov 02 19:43:50.868919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.868899 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:50.918082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.918015 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:50.918391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.918083 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:50.918391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.918251 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:50.918391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.918360 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:50.923171 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:50.923146 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6"] Nov 02 19:43:51.021201 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.021168 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.021402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.021337 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.021464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.021438 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqgz\" (UniqueName: \"kubernetes.io/projected/4185f19e-e5fb-4092-9f86-de99ea5d6304-kube-api-access-qpqgz\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.109131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.109092 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" event={"ID":"772633d4-a7f3-47f3-a51e-fc351e9e366c","Type":"ContainerStarted","Data":"66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e"} Nov 02 19:43:51.109346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.109144 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" event={"ID":"772633d4-a7f3-47f3-a51e-fc351e9e366c","Type":"ContainerStarted","Data":"99808b00d3c41c62a21bdf398c23234f384432851b561e4bb0ac68019103cf46"} Nov 02 19:43:51.123104 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.123028 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqgz\" (UniqueName: \"kubernetes.io/projected/4185f19e-e5fb-4092-9f86-de99ea5d6304-kube-api-access-qpqgz\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.123104 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.123098 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.123291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.123200 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.123944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.123924 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.125639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.125620 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.176774 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.176742 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqgz\" (UniqueName: \"kubernetes.io/projected/4185f19e-e5fb-4092-9f86-de99ea5d6304-kube-api-access-qpqgz\") pod \"maas-api-revoke-sub-17bfcd5e-n6pl6\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.184459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.184431 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:51.220367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.220283 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podStartSLOduration=2.220258681 podStartE2EDuration="2.220258681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:51.217451847 +0000 UTC m=+937.166107685" watchObservedRunningTime="2026-11-02 19:43:51.220258681 +0000 UTC m=+937.168914521" Nov 02 19:43:51.231400 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.231374 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:43:51.560335 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:51.559828 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6"] Nov 02 19:43:52.118862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:52.118766 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" event={"ID":"4185f19e-e5fb-4092-9f86-de99ea5d6304","Type":"ContainerStarted","Data":"7ce7dc8b06985db06b0d56367dec551cb2b4c33dfe987ca850264e3bc607f84d"} Nov 02 19:43:52.118862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:52.118830 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" event={"ID":"4185f19e-e5fb-4092-9f86-de99ea5d6304","Type":"ContainerStarted","Data":"cdec82e7e31483cf7598c2e72165b42c6a8471a9faa5b58bd375c4b510923f8e"} Nov 02 19:43:52.232064 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:52.232012 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" podStartSLOduration=2.231996142 podStartE2EDuration="2.231996142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:52.22534155 +0000 UTC m=+938.173997381" watchObservedRunningTime="2026-11-02 19:43:52.231996142 +0000 UTC m=+938.180651980" Nov 02 19:43:53.127623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:53.127588 4966 generic.go:356] "Generic (PLEG): container finished" podID="4185f19e-e5fb-4092-9f86-de99ea5d6304" containerID="7ce7dc8b06985db06b0d56367dec551cb2b4c33dfe987ca850264e3bc607f84d" exitCode=0 Nov 02 19:43:53.128135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:53.127678 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" event={"ID":"4185f19e-e5fb-4092-9f86-de99ea5d6304","Type":"ContainerDied","Data":"7ce7dc8b06985db06b0d56367dec551cb2b4c33dfe987ca850264e3bc607f84d"} Nov 02 19:43:54.310190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.310164 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:54.357842 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.357789 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4185f19e-e5fb-4092-9f86-de99ea5d6304-kube-api-access-qpqgz\" (UniqueName: \"kubernetes.io/projected/4185f19e-e5fb-4092-9f86-de99ea5d6304-kube-api-access-qpqgz\") pod \"4185f19e-e5fb-4092-9f86-de99ea5d6304\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " Nov 02 19:43:54.359355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.359326 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-xks-ca\") pod \"4185f19e-e5fb-4092-9f86-de99ea5d6304\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " Nov 02 19:43:54.359897 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.359866 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-openshift-ca\") pod \"4185f19e-e5fb-4092-9f86-de99ea5d6304\" (UID: \"4185f19e-e5fb-4092-9f86-de99ea5d6304\") " Nov 02 19:43:54.362166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.360221 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-openshift-ca" pod "4185f19e-e5fb-4092-9f86-de99ea5d6304" (UID: "4185f19e-e5fb-4092-9f86-de99ea5d6304"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:54.362166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.360410 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:54.362405 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.362367 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-xks-ca" pod "4185f19e-e5fb-4092-9f86-de99ea5d6304" (UID: "4185f19e-e5fb-4092-9f86-de99ea5d6304"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:54.362776 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.362751 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4185f19e-e5fb-4092-9f86-de99ea5d6304-kube-api-access-qpqgz" pod "4185f19e-e5fb-4092-9f86-de99ea5d6304" (UID: "4185f19e-e5fb-4092-9f86-de99ea5d6304"). InnerVolumeSpecName "kube-api-access-qpqgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:54.461612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.461541 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4185f19e-e5fb-4092-9f86-de99ea5d6304-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:54.461612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:54.461568 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqgz\" (UniqueName: \"kubernetes.io/projected/4185f19e-e5fb-4092-9f86-de99ea5d6304-kube-api-access-qpqgz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:55.152193 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:55.152155 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" event={"ID":"4185f19e-e5fb-4092-9f86-de99ea5d6304","Type":"ContainerDied","Data":"cdec82e7e31483cf7598c2e72165b42c6a8471a9faa5b58bd375c4b510923f8e"} Nov 02 19:43:55.152484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:55.152462 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdec82e7e31483cf7598c2e72165b42c6a8471a9faa5b58bd375c4b510923f8e" Nov 02 19:43:55.152681 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:55.152666 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6" Nov 02 19:43:56.161275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.161240 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2"] Nov 02 19:43:56.161871 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.161853 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4185f19e-e5fb-4092-9f86-de99ea5d6304" containerName="revoke-keys" Nov 02 19:43:56.161871 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.161871 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4185f19e-e5fb-4092-9f86-de99ea5d6304" containerName="revoke-keys" Nov 02 19:43:56.161998 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.161957 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4185f19e-e5fb-4092-9f86-de99ea5d6304" containerName="revoke-keys" Nov 02 19:43:56.165768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.165744 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.214595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.214565 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:43:56.219353 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.219325 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2"] Nov 02 19:43:56.232843 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.232817 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:43:56.260650 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.260623 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:43:56.260802 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.260651 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:43:56.288569 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.288537 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.288746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.288595 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.288746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.288723 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxx4\" (UniqueName: \"kubernetes.io/projected/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-kube-api-access-wmxx4\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.390174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.390134 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.390444 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.390202 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.390444 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.390262 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxx4\" (UniqueName: \"kubernetes.io/projected/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-kube-api-access-wmxx4\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.391088 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.391064 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.392891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.392868 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.442020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.441936 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f"] Nov 02 19:43:56.445415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.445384 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxx4\" (UniqueName: \"kubernetes.io/projected/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-kube-api-access-wmxx4\") pod \"maas-api-revoke-sub-5178f22d-m5zz2\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.447129 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.447110 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.494315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.493593 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8r4\" (UniqueName: \"kubernetes.io/projected/cb9e08b0-4a06-4872-8591-9c965f2261d5-kube-api-access-dj8r4\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.494315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.493720 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.494315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.493777 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.501797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.501772 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:56.507359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.507331 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f"] Nov 02 19:43:56.595045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.595008 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.595186 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.595071 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.595186 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.595164 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8r4\" (UniqueName: \"kubernetes.io/projected/cb9e08b0-4a06-4872-8591-9c965f2261d5-kube-api-access-dj8r4\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.595896 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.595871 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.597498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.597475 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.647733 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.647707 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8r4\" (UniqueName: \"kubernetes.io/projected/cb9e08b0-4a06-4872-8591-9c965f2261d5-kube-api-access-dj8r4\") pod \"maas-api-revoke-sub-d4a5adb8-bkh6f\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.772644 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.772610 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:56.877065 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:56.877040 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2"] Nov 02 19:43:57.150190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:57.150153 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f"] Nov 02 19:43:57.153124 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:43:57.153089 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9e08b0_4a06_4872_8591_9c965f2261d5.slice/crio-758cbfcb549ea671bf44bbd291a1d14304b1b16e10ac34faf612115c898a41ea WatchSource:0}: Error finding container 758cbfcb549ea671bf44bbd291a1d14304b1b16e10ac34faf612115c898a41ea: Status 404 returned error can't find the container with id 758cbfcb549ea671bf44bbd291a1d14304b1b16e10ac34faf612115c898a41ea Nov 02 19:43:57.170732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:57.170701 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" event={"ID":"cb9e08b0-4a06-4872-8591-9c965f2261d5","Type":"ContainerStarted","Data":"758cbfcb549ea671bf44bbd291a1d14304b1b16e10ac34faf612115c898a41ea"} Nov 02 19:43:57.172590 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:57.172561 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" event={"ID":"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba","Type":"ContainerStarted","Data":"545af9186588c80d02be993527f625b64ba9ed8c3d8a2a7ec3f39cb8dfa2f73b"} Nov 02 19:43:57.172700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:57.172615 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" event={"ID":"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba","Type":"ContainerStarted","Data":"d292697325c19def63cfe1cef614f76154b48f4228732d38905232fdc2120c6b"} Nov 02 19:43:57.279069 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:57.279015 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" podStartSLOduration=1.279000348 podStartE2EDuration="1.279000348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:43:57.276247651 +0000 UTC m=+943.224903472" watchObservedRunningTime="2026-11-02 19:43:57.279000348 +0000 UTC m=+943.227656199" Nov 02 19:43:58.183965 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:58.183875 4966 generic.go:356] "Generic (PLEG): container finished" podID="cb9e08b0-4a06-4872-8591-9c965f2261d5" containerID="410046de65dfce4f1b5cd2bfcb94e3e5e7070b005f4a73b0ebd0cf1a01067bd9" exitCode=0 Nov 02 19:43:58.185223 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:58.185192 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" event={"ID":"cb9e08b0-4a06-4872-8591-9c965f2261d5","Type":"ContainerDied","Data":"410046de65dfce4f1b5cd2bfcb94e3e5e7070b005f4a73b0ebd0cf1a01067bd9"} Nov 02 19:43:58.185817 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:58.185790 4966 generic.go:356] "Generic (PLEG): container finished" podID="9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" containerID="545af9186588c80d02be993527f625b64ba9ed8c3d8a2a7ec3f39cb8dfa2f73b" exitCode=0 Nov 02 19:43:58.185925 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:58.185872 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" event={"ID":"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba","Type":"ContainerDied","Data":"545af9186588c80d02be993527f625b64ba9ed8c3d8a2a7ec3f39cb8dfa2f73b"} Nov 02 19:43:59.360646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.360614 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:43:59.400062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.400039 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:43:59.424483 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424456 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-openshift-ca\") pod \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " Nov 02 19:43:59.424587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424516 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-openshift-ca\") pod \"cb9e08b0-4a06-4872-8591-9c965f2261d5\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " Nov 02 19:43:59.424587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424544 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-xks-ca\") pod \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " Nov 02 19:43:59.424666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424601 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-xks-ca\") pod \"cb9e08b0-4a06-4872-8591-9c965f2261d5\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " Nov 02 19:43:59.424666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424648 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9e08b0-4a06-4872-8591-9c965f2261d5-kube-api-access-dj8r4\" (UniqueName: \"kubernetes.io/projected/cb9e08b0-4a06-4872-8591-9c965f2261d5-kube-api-access-dj8r4\") pod \"cb9e08b0-4a06-4872-8591-9c965f2261d5\" (UID: \"cb9e08b0-4a06-4872-8591-9c965f2261d5\") " Nov 02 19:43:59.424747 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424720 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-kube-api-access-wmxx4\" (UniqueName: \"kubernetes.io/projected/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-kube-api-access-wmxx4\") pod \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\" (UID: \"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba\") " Nov 02 19:43:59.424828 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424803 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-openshift-ca" pod "9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" (UID: "9dbd6486-9d1a-422c-b84a-0ac2b8c99cba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:59.424922 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.424887 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-openshift-ca" pod "cb9e08b0-4a06-4872-8591-9c965f2261d5" (UID: "cb9e08b0-4a06-4872-8591-9c965f2261d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:43:59.425170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.425147 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:59.425253 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.425173 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:59.426812 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.426777 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-xks-ca" pod "9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" (UID: "9dbd6486-9d1a-422c-b84a-0ac2b8c99cba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:59.427057 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.427032 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-kube-api-access-wmxx4" pod "9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" (UID: "9dbd6486-9d1a-422c-b84a-0ac2b8c99cba"). InnerVolumeSpecName "kube-api-access-wmxx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:59.427209 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.427192 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9e08b0-4a06-4872-8591-9c965f2261d5-kube-api-access-dj8r4" pod "cb9e08b0-4a06-4872-8591-9c965f2261d5" (UID: "cb9e08b0-4a06-4872-8591-9c965f2261d5"). InnerVolumeSpecName "kube-api-access-dj8r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:43:59.427209 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.427200 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-xks-ca" pod "cb9e08b0-4a06-4872-8591-9c965f2261d5" (UID: "cb9e08b0-4a06-4872-8591-9c965f2261d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:43:59.526130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.526094 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:59.526130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.526127 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb9e08b0-4a06-4872-8591-9c965f2261d5-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:59.526355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.526139 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj8r4\" (UniqueName: \"kubernetes.io/projected/cb9e08b0-4a06-4872-8591-9c965f2261d5-kube-api-access-dj8r4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:43:59.526355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:43:59.526154 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmxx4\" (UniqueName: \"kubernetes.io/projected/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba-kube-api-access-wmxx4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:00.203187 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.203162 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" Nov 02 19:44:00.203187 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.203175 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f" event={"ID":"cb9e08b0-4a06-4872-8591-9c965f2261d5","Type":"ContainerDied","Data":"758cbfcb549ea671bf44bbd291a1d14304b1b16e10ac34faf612115c898a41ea"} Nov 02 19:44:00.203399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.203229 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758cbfcb549ea671bf44bbd291a1d14304b1b16e10ac34faf612115c898a41ea" Nov 02 19:44:00.204719 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.204700 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" Nov 02 19:44:00.204843 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.204738 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2" event={"ID":"9dbd6486-9d1a-422c-b84a-0ac2b8c99cba","Type":"ContainerDied","Data":"d292697325c19def63cfe1cef614f76154b48f4228732d38905232fdc2120c6b"} Nov 02 19:44:00.204843 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.204821 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d292697325c19def63cfe1cef614f76154b48f4228732d38905232fdc2120c6b" Nov 02 19:44:00.231288 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.231263 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:44:00.244063 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.244041 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:44:00.523728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.523697 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k"] Nov 02 19:44:00.524194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.524179 4966 cpu_manager.go:414] "containerMap: removing container" podUID="9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" containerName="revoke-keys" Nov 02 19:44:00.524237 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.524195 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" containerName="revoke-keys" Nov 02 19:44:00.524237 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.524208 4966 cpu_manager.go:414] "containerMap: removing container" podUID="cb9e08b0-4a06-4872-8591-9c965f2261d5" containerName="revoke-keys" Nov 02 19:44:00.524237 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.524213 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9e08b0-4a06-4872-8591-9c965f2261d5" containerName="revoke-keys" Nov 02 19:44:00.524362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.524349 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" containerName="revoke-keys" Nov 02 19:44:00.524362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.524357 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9e08b0-4a06-4872-8591-9c965f2261d5" containerName="revoke-keys" Nov 02 19:44:00.527790 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.527775 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.579341 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.579307 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:44:00.579512 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.579382 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:44:00.579512 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.579451 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:44:00.579633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.579613 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:44:00.582525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.582498 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k"] Nov 02 19:44:00.639812 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.639775 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.639990 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.639895 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.640055 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.639990 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzlg\" (UniqueName: \"kubernetes.io/projected/292f2ff6-572c-404d-bcb0-af60496177c5-kube-api-access-wkzlg\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.740730 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.740695 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.740939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.740764 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzlg\" (UniqueName: \"kubernetes.io/projected/292f2ff6-572c-404d-bcb0-af60496177c5-kube-api-access-wkzlg\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.740939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.740892 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.741508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.741482 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.743352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.743325 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.794523 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.794455 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzlg\" (UniqueName: \"kubernetes.io/projected/292f2ff6-572c-404d-bcb0-af60496177c5-kube-api-access-wkzlg\") pod \"maas-api-revoke-sub-2e331cba-5l25k\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:00.841538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:00.841508 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:01.216168 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:01.216135 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k"] Nov 02 19:44:01.216483 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:44:01.216453 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292f2ff6_572c_404d_bcb0_af60496177c5.slice/crio-1ed3e93b786037b7781a1f20fcbc809131962c5ac93faef9df386ea44fda78d6 WatchSource:0}: Error finding container 1ed3e93b786037b7781a1f20fcbc809131962c5ac93faef9df386ea44fda78d6: Status 404 returned error can't find the container with id 1ed3e93b786037b7781a1f20fcbc809131962c5ac93faef9df386ea44fda78d6 Nov 02 19:44:01.225536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:01.225509 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:44:02.225050 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:02.225014 4966 generic.go:356] "Generic (PLEG): container finished" podID="292f2ff6-572c-404d-bcb0-af60496177c5" containerID="3668f5a1bb104627fff0d6ba77abcfb834cde3320b71f7fc1d8d8c3e911e275d" exitCode=0 Nov 02 19:44:02.225514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:02.225124 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" event={"ID":"292f2ff6-572c-404d-bcb0-af60496177c5","Type":"ContainerDied","Data":"3668f5a1bb104627fff0d6ba77abcfb834cde3320b71f7fc1d8d8c3e911e275d"} Nov 02 19:44:02.225514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:02.225188 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" event={"ID":"292f2ff6-572c-404d-bcb0-af60496177c5","Type":"ContainerStarted","Data":"1ed3e93b786037b7781a1f20fcbc809131962c5ac93faef9df386ea44fda78d6"} Nov 02 19:44:03.396030 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.396003 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:03.473021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.472989 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-xks-ca\") pod \"292f2ff6-572c-404d-bcb0-af60496177c5\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " Nov 02 19:44:03.473162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.473096 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292f2ff6-572c-404d-bcb0-af60496177c5-kube-api-access-wkzlg\" (UniqueName: \"kubernetes.io/projected/292f2ff6-572c-404d-bcb0-af60496177c5-kube-api-access-wkzlg\") pod \"292f2ff6-572c-404d-bcb0-af60496177c5\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " Nov 02 19:44:03.473162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.473147 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-openshift-ca\") pod \"292f2ff6-572c-404d-bcb0-af60496177c5\" (UID: \"292f2ff6-572c-404d-bcb0-af60496177c5\") " Nov 02 19:44:03.473593 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.473567 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-openshift-ca" pod "292f2ff6-572c-404d-bcb0-af60496177c5" (UID: "292f2ff6-572c-404d-bcb0-af60496177c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:44:03.475062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.475034 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-xks-ca" pod "292f2ff6-572c-404d-bcb0-af60496177c5" (UID: "292f2ff6-572c-404d-bcb0-af60496177c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:44:03.475187 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.475163 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292f2ff6-572c-404d-bcb0-af60496177c5-kube-api-access-wkzlg" pod "292f2ff6-572c-404d-bcb0-af60496177c5" (UID: "292f2ff6-572c-404d-bcb0-af60496177c5"). InnerVolumeSpecName "kube-api-access-wkzlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:44:03.574706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.574672 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:03.574706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.574702 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzlg\" (UniqueName: \"kubernetes.io/projected/292f2ff6-572c-404d-bcb0-af60496177c5-kube-api-access-wkzlg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:03.574706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:03.574713 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292f2ff6-572c-404d-bcb0-af60496177c5-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:04.240402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:04.240371 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" event={"ID":"292f2ff6-572c-404d-bcb0-af60496177c5","Type":"ContainerDied","Data":"1ed3e93b786037b7781a1f20fcbc809131962c5ac93faef9df386ea44fda78d6"} Nov 02 19:44:04.240402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:04.240398 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k" Nov 02 19:44:04.240609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:04.240419 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed3e93b786037b7781a1f20fcbc809131962c5ac93faef9df386ea44fda78d6" Nov 02 19:44:14.754447 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:44:14.754400 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:44:15.690280 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.690242 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g"] Nov 02 19:44:15.690881 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.690862 4966 cpu_manager.go:414] "containerMap: removing container" podUID="292f2ff6-572c-404d-bcb0-af60496177c5" containerName="revoke-keys" Nov 02 19:44:15.690975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.690882 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292f2ff6-572c-404d-bcb0-af60496177c5" containerName="revoke-keys" Nov 02 19:44:15.691041 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.691019 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="292f2ff6-572c-404d-bcb0-af60496177c5" containerName="revoke-keys" Nov 02 19:44:15.695632 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.695610 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.750679 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.750527 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:44:15.750679 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.750667 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:44:15.751359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.751334 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:44:15.751473 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.751441 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:44:15.752327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.752277 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g"] Nov 02 19:44:15.808814 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.808778 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.809266 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.808885 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cpvk\" (UniqueName: \"kubernetes.io/projected/5255ae7e-ac4e-44f8-904f-9e5796b65f33-kube-api-access-9cpvk\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.809266 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.808956 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.910040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.910010 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.910194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.910072 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cpvk\" (UniqueName: \"kubernetes.io/projected/5255ae7e-ac4e-44f8-904f-9e5796b65f33-kube-api-access-9cpvk\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.910194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.910117 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.910778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.910749 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.912710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.912680 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:15.964234 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:15.964163 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cpvk\" (UniqueName: \"kubernetes.io/projected/5255ae7e-ac4e-44f8-904f-9e5796b65f33-kube-api-access-9cpvk\") pod \"maas-api-revoke-sub-57bc21bd-zzm6g\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:16.009896 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:16.009863 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:16.378612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:16.378579 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g"] Nov 02 19:44:16.379938 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:44:16.379910 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5255ae7e_ac4e_44f8_904f_9e5796b65f33.slice/crio-574eda424882cce8e4a9624da2a093e69d787b53ee89dcb72b115ac471581aa1 WatchSource:0}: Error finding container 574eda424882cce8e4a9624da2a093e69d787b53ee89dcb72b115ac471581aa1: Status 404 returned error can't find the container with id 574eda424882cce8e4a9624da2a093e69d787b53ee89dcb72b115ac471581aa1 Nov 02 19:44:17.349230 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:17.349191 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" event={"ID":"5255ae7e-ac4e-44f8-904f-9e5796b65f33","Type":"ContainerStarted","Data":"d0e11db5fa00faa2cfc16f37dbee72eb02b14809415ca3a141d78ea6f82e1ae1"} Nov 02 19:44:17.349674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:17.349257 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" event={"ID":"5255ae7e-ac4e-44f8-904f-9e5796b65f33","Type":"ContainerStarted","Data":"574eda424882cce8e4a9624da2a093e69d787b53ee89dcb72b115ac471581aa1"} Nov 02 19:44:17.458482 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:17.458429 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" podStartSLOduration=2.458415275 podStartE2EDuration="2.458415275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:44:17.455291495 +0000 UTC m=+963.403947335" watchObservedRunningTime="2026-11-02 19:44:17.458415275 +0000 UTC m=+963.407071115" Nov 02 19:44:18.361161 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.361126 4966 generic.go:356] "Generic (PLEG): container finished" podID="5255ae7e-ac4e-44f8-904f-9e5796b65f33" containerID="d0e11db5fa00faa2cfc16f37dbee72eb02b14809415ca3a141d78ea6f82e1ae1" exitCode=0 Nov 02 19:44:18.361696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.361209 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" event={"ID":"5255ae7e-ac4e-44f8-904f-9e5796b65f33","Type":"ContainerDied","Data":"d0e11db5fa00faa2cfc16f37dbee72eb02b14809415ca3a141d78ea6f82e1ae1"} Nov 02 19:44:18.464448 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.464409 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh"] Nov 02 19:44:18.468698 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.468670 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.510351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.510315 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh"] Nov 02 19:44:18.520610 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.520579 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5c46af-openshift-default-dockercfg-c97j7\"" Nov 02 19:44:18.542423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.542376 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.542789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.542759 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc150de8-bfd9-468d-b366-91177eadded8-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.543009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.542971 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.543236 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.543215 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4k8v\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-kube-api-access-s4k8v\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.543506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.543486 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.543715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.543698 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc150de8-bfd9-468d-b366-91177eadded8-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.543909 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.543893 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.544117 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.544102 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.544274 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.544252 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646293 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646204 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646293 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646267 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc150de8-bfd9-468d-b366-91177eadded8-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646350 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646415 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646438 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646504 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646571 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc150de8-bfd9-468d-b366-91177eadded8-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646597 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646910 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646665 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4k8v\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-kube-api-access-s4k8v\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.646910 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.646727 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.647235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.647203 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.647393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.647318 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.647536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.647513 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.648052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.648029 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc150de8-bfd9-468d-b366-91177eadded8-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.649401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.649363 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.650111 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.650088 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc150de8-bfd9-468d-b366-91177eadded8-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.700834 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.700798 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.701036 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.701009 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4k8v\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-kube-api-access-s4k8v\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:18.785112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:18.785071 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:19.090160 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.090130 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4"] Nov 02 19:44:19.095604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.095583 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.145227 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.145190 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1d869e-openshift-default-dockercfg-cg57h\"" Nov 02 19:44:19.152221 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.152183 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4"] Nov 02 19:44:19.245372 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:44:19.245336 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc150de8_bfd9_468d_b366_91177eadded8.slice/crio-a7e0a1a3c5316e68f15a6c21ad8822d30481a8130e683088b0d6925b2179e86c WatchSource:0}: Error finding container a7e0a1a3c5316e68f15a6c21ad8822d30481a8130e683088b0d6925b2179e86c: Status 404 returned error can't find the container with id a7e0a1a3c5316e68f15a6c21ad8822d30481a8130e683088b0d6925b2179e86c Nov 02 19:44:19.246372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.246347 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh"] Nov 02 19:44:19.248203 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.248167 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:44:19.248321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.248237 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:44:19.248397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.248291 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:44:19.266841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.266807 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.266942 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.266890 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.267023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.266941 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.267023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.266973 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.267137 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.267032 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.267137 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.267104 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.267244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.267137 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrm4\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-kube-api-access-jvrm4\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.267244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.267168 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.267244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.267236 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.372003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.370053 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.372003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.370124 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.372003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.370186 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.372003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.370225 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.372003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.371097 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.372003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.371629 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.377251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.376742 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.377251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.377036 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.377464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.377250 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.377464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.377394 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.377464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.377445 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrm4\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-kube-api-access-jvrm4\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.377635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.377484 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.377635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.377542 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.385397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.378344 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.385397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.378623 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.391664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.391629 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" event={"ID":"cc150de8-bfd9-468d-b366-91177eadded8","Type":"ContainerStarted","Data":"5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25"} Nov 02 19:44:19.391802 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.391736 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" event={"ID":"cc150de8-bfd9-468d-b366-91177eadded8","Type":"ContainerStarted","Data":"a7e0a1a3c5316e68f15a6c21ad8822d30481a8130e683088b0d6925b2179e86c"} Nov 02 19:44:19.393973 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.393950 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.430907 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.430876 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrm4\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-kube-api-access-jvrm4\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.431049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.430952 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.506165 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.506111 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podStartSLOduration=1.506096587 podStartE2EDuration="1.506096587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:44:19.502790411 +0000 UTC m=+965.451446253" watchObservedRunningTime="2026-11-02 19:44:19.506096587 +0000 UTC m=+965.454752426" Nov 02 19:44:19.567284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.567255 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:19.687113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.687082 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255ae7e-ac4e-44f8-904f-9e5796b65f33-kube-api-access-9cpvk\" (UniqueName: \"kubernetes.io/projected/5255ae7e-ac4e-44f8-904f-9e5796b65f33-kube-api-access-9cpvk\") pod \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " Nov 02 19:44:19.687359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.687338 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-xks-ca\") pod \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " Nov 02 19:44:19.687535 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.687517 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-openshift-ca\") pod \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\" (UID: \"5255ae7e-ac4e-44f8-904f-9e5796b65f33\") " Nov 02 19:44:19.688226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.688201 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-openshift-ca" pod "5255ae7e-ac4e-44f8-904f-9e5796b65f33" (UID: "5255ae7e-ac4e-44f8-904f-9e5796b65f33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:44:19.689941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.689918 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-xks-ca" pod "5255ae7e-ac4e-44f8-904f-9e5796b65f33" (UID: "5255ae7e-ac4e-44f8-904f-9e5796b65f33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:44:19.690416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.690388 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255ae7e-ac4e-44f8-904f-9e5796b65f33-kube-api-access-9cpvk" pod "5255ae7e-ac4e-44f8-904f-9e5796b65f33" (UID: "5255ae7e-ac4e-44f8-904f-9e5796b65f33"). InnerVolumeSpecName "kube-api-access-9cpvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:44:19.709575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.709550 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:19.787439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.787405 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:19.789495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.789464 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:19.789580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.789502 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5255ae7e-ac4e-44f8-904f-9e5796b65f33-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:19.789580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:19.789524 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cpvk\" (UniqueName: \"kubernetes.io/projected/5255ae7e-ac4e-44f8-904f-9e5796b65f33-kube-api-access-9cpvk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:20.146078 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.146018 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4"] Nov 02 19:44:20.172223 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.172187 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:44:20.172403 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.172277 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:44:20.172403 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.172350 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:44:20.402243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.401527 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" Nov 02 19:44:20.402243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.401791 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g" event={"ID":"5255ae7e-ac4e-44f8-904f-9e5796b65f33","Type":"ContainerDied","Data":"574eda424882cce8e4a9624da2a093e69d787b53ee89dcb72b115ac471581aa1"} Nov 02 19:44:20.402243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.401846 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574eda424882cce8e4a9624da2a093e69d787b53ee89dcb72b115ac471581aa1" Nov 02 19:44:20.405241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.405219 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" event={"ID":"31e54d54-22dc-48f5-91b2-dc71a057d0e9","Type":"ContainerStarted","Data":"7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8"} Nov 02 19:44:20.405686 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.405651 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" event={"ID":"31e54d54-22dc-48f5-91b2-dc71a057d0e9","Type":"ContainerStarted","Data":"854b6030ec34a14e78d61bba2dadd9a88d079efd62f2cdd3e8a7f5b141c18127"} Nov 02 19:44:20.522533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.522470 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podStartSLOduration=1.522453038 podStartE2EDuration="1.522453038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:44:20.518898588 +0000 UTC m=+966.467554426" watchObservedRunningTime="2026-11-02 19:44:20.522453038 +0000 UTC m=+966.471108901" Nov 02 19:44:20.710331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.710234 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:20.787642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.787580 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 19:44:20.788015 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:20.787981 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 19:44:21.711534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:21.711503 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 19:44:21.712007 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:21.711554 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 19:44:21.786355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:21.785690 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 19:44:21.786355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:21.785759 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 19:44:22.710822 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:22.710783 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 19:44:22.710999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:22.710836 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 19:44:22.786885 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:22.786833 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 19:44:22.787574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:22.787520 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 19:44:23.710961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:23.710922 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 19:44:23.711137 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:23.710981 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 19:44:23.788922 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:23.788888 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 19:44:23.789369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:23.788979 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 19:44:24.710130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:24.710078 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 19:44:24.710368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:24.710139 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 19:44:24.786261 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:24.786219 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 19:44:24.786671 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:24.786636 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 19:44:25.710596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:25.710563 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 19:44:25.711135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:25.710623 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 19:44:25.786088 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:25.786047 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 19:44:25.786291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:25.786110 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 19:44:26.710583 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:26.710548 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 19:44:26.710754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:26.710604 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 19:44:26.790707 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:26.790672 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:26.791083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:26.791058 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:26.791841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:26.791820 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 19:44:27.714469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:27.714438 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:27.714959 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:27.714814 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:27.715507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:27.715486 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 19:44:44.764317 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:44:44.764247 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:44:49.494746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:49.494626 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc"] Nov 02 19:44:49.502040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:49.494828 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" containerID="cri-o://dd97edb115f57ddb7519669153e07d01752fe8c1b46b31803fa12747bf544328" gracePeriod=30 Nov 02 19:44:54.716258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.716225 4966 generic.go:356] "Generic (PLEG): container finished" podID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerID="dd97edb115f57ddb7519669153e07d01752fe8c1b46b31803fa12747bf544328" exitCode=0 Nov 02 19:44:54.716822 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.716285 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" event={"ID":"e9671ce7-e5a4-4f0b-a6df-929cafc909a5","Type":"ContainerDied","Data":"dd97edb115f57ddb7519669153e07d01752fe8c1b46b31803fa12747bf544328"} Nov 02 19:44:54.764426 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.764363 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:44:54.838533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838502 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-socket\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.838706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838600 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-certs\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.838706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838623 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-data\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.838706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838696 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-kube-api-access-kjjnp\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-kube-api-access-kjjnp\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.838850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838725 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-podinfo\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.838850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838803 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-credential-socket\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.838956 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838874 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-envoy\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.838956 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838903 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-token\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-token\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.839027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838913 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-socket" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:44:54.839027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838927 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-certs" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:44:54.839027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.838960 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istiod-ca-cert\") pod \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\" (UID: \"e9671ce7-e5a4-4f0b-a6df-929cafc909a5\") " Nov 02 19:44:54.839249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839223 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-data" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:44:54.839429 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839403 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-credential-socket" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:44:54.839511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839471 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istiod-ca-cert" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:44:54.839608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839590 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.839680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839615 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.839680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839629 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.839680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839644 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.839680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.839658 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.841550 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.841522 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-kube-api-access-kjjnp" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "kube-api-access-kjjnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:44:54.841659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.841591 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-token" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:44:54.841751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.841733 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-envoy" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:44:54.841832 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.841811 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-podinfo" pod "e9671ce7-e5a4-4f0b-a6df-929cafc909a5" (UID: "e9671ce7-e5a4-4f0b-a6df-929cafc909a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:44:54.941283 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.941253 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjjnp\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-kube-api-access-kjjnp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.941283 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.941283 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.941489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.941294 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:54.941489 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:54.941318 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9671ce7-e5a4-4f0b-a6df-929cafc909a5-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:44:55.725453 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:55.725423 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" Nov 02 19:44:55.725911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:55.725457 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc" event={"ID":"e9671ce7-e5a4-4f0b-a6df-929cafc909a5","Type":"ContainerDied","Data":"45a80be09a0dc29f24226636becfc7f7e3fee6f0e65b94b277c61f78a41c1e10"} Nov 02 19:44:55.725911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:55.725522 4966 scope.go:122] "RemoveContainer" containerID="dd97edb115f57ddb7519669153e07d01752fe8c1b46b31803fa12747bf544328" Nov 02 19:44:55.928077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:55.928032 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc"] Nov 02 19:44:55.932170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:55.932144 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-rzqnc"] Nov 02 19:44:56.187629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:44:56.187547 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" path="/var/lib/kubelet/pods/e9671ce7-e5a4-4f0b-a6df-929cafc909a5/volumes" Nov 02 19:45:00.223789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.223754 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp"] Nov 02 19:45:00.224328 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.224312 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" Nov 02 19:45:00.224328 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.224328 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" Nov 02 19:45:00.224462 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.224337 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5255ae7e-ac4e-44f8-904f-9e5796b65f33" containerName="revoke-keys" Nov 02 19:45:00.224462 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.224342 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5255ae7e-ac4e-44f8-904f-9e5796b65f33" containerName="revoke-keys" Nov 02 19:45:00.224462 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.224424 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9671ce7-e5a4-4f0b-a6df-929cafc909a5" containerName="istio-proxy" Nov 02 19:45:00.224462 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.224433 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5255ae7e-ac4e-44f8-904f-9e5796b65f33" containerName="revoke-keys" Nov 02 19:45:00.227804 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.227785 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2"] Nov 02 19:45:00.227937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.227914 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.231657 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.231638 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42"] Nov 02 19:45:00.231752 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.231736 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.235471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.235424 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.272287 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.272258 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77"] Nov 02 19:45:00.276971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.276951 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.281744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.281721 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:45:00.282913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.282893 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:45:00.296569 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.296545 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ac8ee9-251d-4297-88de-ff5009bef651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.296680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.296606 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6gb\" (UniqueName: \"kubernetes.io/projected/194b6c77-9946-40b4-b587-9ff7502dc099-kube-api-access-hw6gb\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.296680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.296654 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.296795 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.296728 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcp2\" (UniqueName: \"kubernetes.io/projected/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-kube-api-access-8fcp2\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.296854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.296842 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2ln\" (UniqueName: \"kubernetes.io/projected/18ac8ee9-251d-4297-88de-ff5009bef651-kube-api-access-8q2ln\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.296904 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.296891 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194b6c77-9946-40b4-b587-9ff7502dc099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.311523 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.311497 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:45:00.331389 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.331364 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt"] Nov 02 19:45:00.335485 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.335465 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2"] Nov 02 19:45:00.335605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.335549 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.340760 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.340076 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp"] Nov 02 19:45:00.345468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.345447 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42"] Nov 02 19:45:00.370320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.370271 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt"] Nov 02 19:45:00.383763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.383730 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77"] Nov 02 19:45:00.398548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.398510 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nj66\" (UniqueName: \"kubernetes.io/projected/e1eb5984-46ee-450f-a991-76f890b6fc17-kube-api-access-9nj66\") pod \"maas-api-key-cleanup-29894145-5gt77\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.398739 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.398559 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlr8\" (UniqueName: \"kubernetes.io/projected/f194e2b3-0861-48f0-869e-c7e76883d9ef-kube-api-access-4tlr8\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.398739 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.398599 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6gb\" (UniqueName: \"kubernetes.io/projected/194b6c77-9946-40b4-b587-9ff7502dc099-kube-api-access-hw6gb\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.398864 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.398746 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.398864 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.398804 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcp2\" (UniqueName: \"kubernetes.io/projected/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-kube-api-access-8fcp2\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.399033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.398905 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f194e2b3-0861-48f0-869e-c7e76883d9ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.399033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.398986 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2ln\" (UniqueName: \"kubernetes.io/projected/18ac8ee9-251d-4297-88de-ff5009bef651-kube-api-access-8q2ln\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.399149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.399068 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194b6c77-9946-40b4-b587-9ff7502dc099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.399149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.399101 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eb5984-46ee-450f-a991-76f890b6fc17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894145-5gt77\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.399149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.399139 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ac8ee9-251d-4297-88de-ff5009bef651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.399737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.399708 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.399984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.399963 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194b6c77-9946-40b4-b587-9ff7502dc099-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.400047 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.400009 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ac8ee9-251d-4297-88de-ff5009bef651-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.451610 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.451581 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6gb\" (UniqueName: \"kubernetes.io/projected/194b6c77-9946-40b4-b587-9ff7502dc099-kube-api-access-hw6gb\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.451919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.451898 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcp2\" (UniqueName: \"kubernetes.io/projected/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-kube-api-access-8fcp2\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.452007 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.451971 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2ln\" (UniqueName: \"kubernetes.io/projected/18ac8ee9-251d-4297-88de-ff5009bef651-kube-api-access-8q2ln\") pod \"maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.500741 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.500668 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f194e2b3-0861-48f0-869e-c7e76883d9ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.500891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.500761 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eb5984-46ee-450f-a991-76f890b6fc17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894145-5gt77\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.500891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.500809 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nj66\" (UniqueName: \"kubernetes.io/projected/e1eb5984-46ee-450f-a991-76f890b6fc17-kube-api-access-9nj66\") pod \"maas-api-key-cleanup-29894145-5gt77\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.500891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.500824 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlr8\" (UniqueName: \"kubernetes.io/projected/f194e2b3-0861-48f0-869e-c7e76883d9ef-kube-api-access-4tlr8\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.501327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.501279 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f194e2b3-0861-48f0-869e-c7e76883d9ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.501402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.501365 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eb5984-46ee-450f-a991-76f890b6fc17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894145-5gt77\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.544890 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.544869 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:00.554516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.554486 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlr8\" (UniqueName: \"kubernetes.io/projected/f194e2b3-0861-48f0-869e-c7e76883d9ef-kube-api-access-4tlr8\") pod \"maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.554800 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.554774 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nj66\" (UniqueName: \"kubernetes.io/projected/e1eb5984-46ee-450f-a991-76f890b6fc17-kube-api-access-9nj66\") pod \"maas-api-key-cleanup-29894145-5gt77\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.564485 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.564446 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:00.576798 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.576776 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:00.590982 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.590797 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:00.670850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.659837 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:00.927873 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.927845 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp"] Nov 02 19:45:00.928675 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:45:00.928649 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ac8ee9_251d_4297_88de_ff5009bef651.slice/crio-fc205962627da6430b62b3e3bffb88a868b123fd9d1f3a4ae959f7637476168b WatchSource:0}: Error finding container fc205962627da6430b62b3e3bffb88a868b123fd9d1f3a4ae959f7637476168b: Status 404 returned error can't find the container with id fc205962627da6430b62b3e3bffb88a868b123fd9d1f3a4ae959f7637476168b Nov 02 19:45:00.975261 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:45:00.975231 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625c5b37_0ad8_4c56_9e72_04a8c5bfb457.slice/crio-df674f0bc3b1cd6e4ea2e2c42fb9dc50dd4ac7dc6e581a60f7be7cf45727fab4 WatchSource:0}: Error finding container df674f0bc3b1cd6e4ea2e2c42fb9dc50dd4ac7dc6e581a60f7be7cf45727fab4: Status 404 returned error can't find the container with id df674f0bc3b1cd6e4ea2e2c42fb9dc50dd4ac7dc6e581a60f7be7cf45727fab4 Nov 02 19:45:00.978192 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:00.978169 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2"] Nov 02 19:45:01.012158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.012104 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42"] Nov 02 19:45:01.013514 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:45:01.013492 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194b6c77_9946_40b4_b587_9ff7502dc099.slice/crio-2e0365d5ff009f8571277fd579b989888a2b4c5848136c1d0024a90e03a8ce74 WatchSource:0}: Error finding container 2e0365d5ff009f8571277fd579b989888a2b4c5848136c1d0024a90e03a8ce74: Status 404 returned error can't find the container with id 2e0365d5ff009f8571277fd579b989888a2b4c5848136c1d0024a90e03a8ce74 Nov 02 19:45:01.023475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.023454 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77"] Nov 02 19:45:01.025927 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:45:01.025904 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1eb5984_46ee_450f_a991_76f890b6fc17.slice/crio-c61b35511b82e4b8933ee18879216c0496aa8457260b35f1d3f32f1eda42ee88 WatchSource:0}: Error finding container c61b35511b82e4b8933ee18879216c0496aa8457260b35f1d3f32f1eda42ee88: Status 404 returned error can't find the container with id c61b35511b82e4b8933ee18879216c0496aa8457260b35f1d3f32f1eda42ee88 Nov 02 19:45:01.095173 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.095141 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt"] Nov 02 19:45:01.146493 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:45:01.146458 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf194e2b3_0861_48f0_869e_c7e76883d9ef.slice/crio-4714d764084dc60f23b6bbc923616f2a059f9392cb25eea82fea79e97271243e WatchSource:0}: Error finding container 4714d764084dc60f23b6bbc923616f2a059f9392cb25eea82fea79e97271243e: Status 404 returned error can't find the container with id 4714d764084dc60f23b6bbc923616f2a059f9392cb25eea82fea79e97271243e Nov 02 19:45:01.787206 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.787166 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" event={"ID":"f194e2b3-0861-48f0-869e-c7e76883d9ef","Type":"ContainerStarted","Data":"ffcde33b91eb7d6276f8601f1f0e96882577b5c912e19df6df1f56e377c04934"} Nov 02 19:45:01.787733 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.787230 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" event={"ID":"f194e2b3-0861-48f0-869e-c7e76883d9ef","Type":"ContainerStarted","Data":"4714d764084dc60f23b6bbc923616f2a059f9392cb25eea82fea79e97271243e"} Nov 02 19:45:01.789047 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.789020 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerStarted","Data":"0b712f823cff7595355968c7cbcd8046dc7b1154299c451e6b61493f9c758b78"} Nov 02 19:45:01.789159 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.789074 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerStarted","Data":"fc205962627da6430b62b3e3bffb88a868b123fd9d1f3a4ae959f7637476168b"} Nov 02 19:45:01.790652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.790624 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerStarted","Data":"43b865c840a93405df4f596b7256ebf701f71fde9b7af0bd5ee4ddf93c2f73dc"} Nov 02 19:45:01.790757 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.790675 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerStarted","Data":"df674f0bc3b1cd6e4ea2e2c42fb9dc50dd4ac7dc6e581a60f7be7cf45727fab4"} Nov 02 19:45:01.792279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.792256 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" event={"ID":"194b6c77-9946-40b4-b587-9ff7502dc099","Type":"ContainerStarted","Data":"ed8ead9e726034796ef3d560422a2090fd0ba5381c05c688bef69246c368a90f"} Nov 02 19:45:01.792385 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.792311 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" event={"ID":"194b6c77-9946-40b4-b587-9ff7502dc099","Type":"ContainerStarted","Data":"2e0365d5ff009f8571277fd579b989888a2b4c5848136c1d0024a90e03a8ce74"} Nov 02 19:45:01.793777 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.793757 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerStarted","Data":"90aef27334a8ef03a74bbb5fff550d4390fb984f0b90fc422e867767eaaac4f4"} Nov 02 19:45:01.793850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.793802 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerStarted","Data":"c61b35511b82e4b8933ee18879216c0496aa8457260b35f1d3f32f1eda42ee88"} Nov 02 19:45:01.894691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.894638 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" podStartSLOduration=1.8946220409999999 podStartE2EDuration="1.894622041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:01.890597418 +0000 UTC m=+1007.839253260" watchObservedRunningTime="2026-11-02 19:45:01.894622041 +0000 UTC m=+1007.843277888" Nov 02 19:45:01.999588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:01.999534 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" podStartSLOduration=1.999517847 podStartE2EDuration="1.999517847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:01.995459771 +0000 UTC m=+1007.944115611" watchObservedRunningTime="2026-11-02 19:45:01.999517847 +0000 UTC m=+1007.948173685" Nov 02 19:45:02.102622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.102557 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" podStartSLOduration=2.102537689 podStartE2EDuration="2.102537689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:02.100837638 +0000 UTC m=+1008.049493478" watchObservedRunningTime="2026-11-02 19:45:02.102537689 +0000 UTC m=+1008.051193529" Nov 02 19:45:02.207713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.207598 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" podStartSLOduration=2.207583741 podStartE2EDuration="2.207583741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:02.205513679 +0000 UTC m=+1008.154169518" watchObservedRunningTime="2026-11-02 19:45:02.207583741 +0000 UTC m=+1008.156239580" Nov 02 19:45:02.313775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.313721 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" podStartSLOduration=2.313707277 podStartE2EDuration="2.313707277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:02.309812463 +0000 UTC m=+1008.258468296" watchObservedRunningTime="2026-11-02 19:45:02.313707277 +0000 UTC m=+1008.262363116" Nov 02 19:45:02.803060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.803026 4966 generic.go:356] "Generic (PLEG): container finished" podID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerID="90aef27334a8ef03a74bbb5fff550d4390fb984f0b90fc422e867767eaaac4f4" exitCode=60 Nov 02 19:45:02.803595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.803117 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerDied","Data":"90aef27334a8ef03a74bbb5fff550d4390fb984f0b90fc422e867767eaaac4f4"} Nov 02 19:45:02.803595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.803400 4966 scope.go:122] "RemoveContainer" containerID="90aef27334a8ef03a74bbb5fff550d4390fb984f0b90fc422e867767eaaac4f4" Nov 02 19:45:02.805162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.805097 4966 generic.go:356] "Generic (PLEG): container finished" podID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerID="ffcde33b91eb7d6276f8601f1f0e96882577b5c912e19df6df1f56e377c04934" exitCode=60 Nov 02 19:45:02.805244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.805159 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" event={"ID":"f194e2b3-0861-48f0-869e-c7e76883d9ef","Type":"ContainerDied","Data":"ffcde33b91eb7d6276f8601f1f0e96882577b5c912e19df6df1f56e377c04934"} Nov 02 19:45:02.805497 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.805479 4966 scope.go:122] "RemoveContainer" containerID="ffcde33b91eb7d6276f8601f1f0e96882577b5c912e19df6df1f56e377c04934" Nov 02 19:45:02.807112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.807086 4966 generic.go:356] "Generic (PLEG): container finished" podID="18ac8ee9-251d-4297-88de-ff5009bef651" containerID="0b712f823cff7595355968c7cbcd8046dc7b1154299c451e6b61493f9c758b78" exitCode=60 Nov 02 19:45:02.807256 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.807228 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerDied","Data":"0b712f823cff7595355968c7cbcd8046dc7b1154299c451e6b61493f9c758b78"} Nov 02 19:45:02.807403 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.807375 4966 scope.go:122] "RemoveContainer" containerID="0b712f823cff7595355968c7cbcd8046dc7b1154299c451e6b61493f9c758b78" Nov 02 19:45:02.809247 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.809224 4966 generic.go:356] "Generic (PLEG): container finished" podID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerID="43b865c840a93405df4f596b7256ebf701f71fde9b7af0bd5ee4ddf93c2f73dc" exitCode=60 Nov 02 19:45:02.809396 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.809293 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerDied","Data":"43b865c840a93405df4f596b7256ebf701f71fde9b7af0bd5ee4ddf93c2f73dc"} Nov 02 19:45:02.809514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.809497 4966 scope.go:122] "RemoveContainer" containerID="43b865c840a93405df4f596b7256ebf701f71fde9b7af0bd5ee4ddf93c2f73dc" Nov 02 19:45:02.811114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.811088 4966 generic.go:356] "Generic (PLEG): container finished" podID="194b6c77-9946-40b4-b587-9ff7502dc099" containerID="ed8ead9e726034796ef3d560422a2090fd0ba5381c05c688bef69246c368a90f" exitCode=60 Nov 02 19:45:02.811212 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.811166 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" event={"ID":"194b6c77-9946-40b4-b587-9ff7502dc099","Type":"ContainerDied","Data":"ed8ead9e726034796ef3d560422a2090fd0ba5381c05c688bef69246c368a90f"} Nov 02 19:45:02.811356 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:02.811341 4966 scope.go:122] "RemoveContainer" containerID="ed8ead9e726034796ef3d560422a2090fd0ba5381c05c688bef69246c368a90f" Nov 02 19:45:03.117349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.117313 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5"] Nov 02 19:45:03.126163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.126135 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.178316 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.178276 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-43ceaa-openshift-default-dockercfg-nlhql\"" Nov 02 19:45:03.181257 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.181230 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5"] Nov 02 19:45:03.233649 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.233613 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca23981-937a-43ec-8571-065f11cd2175-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.233829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.233657 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.233829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.233781 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.233910 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.233827 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngc5b\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-kube-api-access-ngc5b\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.233954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.233927 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.234035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.234023 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca23981-937a-43ec-8571-065f11cd2175-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.234071 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.234045 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.234115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.234078 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.234147 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.234140 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.335796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.335713 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.335796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.335761 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca23981-937a-43ec-8571-065f11cd2175-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.335800 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.335861 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.335890 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngc5b\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-kube-api-access-ngc5b\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.335948 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.336028 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca23981-937a-43ec-8571-065f11cd2175-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.336056 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.336120 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.336204 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.336411 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.336448 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.336688 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.336661 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.337189 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.337164 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca23981-937a-43ec-8571-065f11cd2175-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.346399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.346376 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca23981-937a-43ec-8571-065f11cd2175-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.346399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.346385 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.389455 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.389422 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.389917 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.389896 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngc5b\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-kube-api-access-ngc5b\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.442513 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.442479 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:03.822713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.822669 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" event={"ID":"f194e2b3-0861-48f0-869e-c7e76883d9ef","Type":"ContainerStarted","Data":"82bc20d3d61075ff7a10dfb0bd700c8cfe2dc4d687779104522abef8082a0c06"} Nov 02 19:45:03.825079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.825050 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerStarted","Data":"b975feea70aaf9eea2056e2f24319f1398bd6ac797bcc49fb6abb0360e3e13cd"} Nov 02 19:45:03.827198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.827170 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerStarted","Data":"6198092bea8b1fa1d503df41f51dc21349e8c9f75c5fdf84d9cb165d6b5bb82a"} Nov 02 19:45:03.829265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.829241 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" event={"ID":"194b6c77-9946-40b4-b587-9ff7502dc099","Type":"ContainerStarted","Data":"2949cba05512acb6d17e16e211b7845361519cbf6dcc279292180750f65d7094"} Nov 02 19:45:03.831268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.831249 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerStarted","Data":"3fe4850c4dade713c42793c0924041ec804cace6b392341d53ae9b386a0349c5"} Nov 02 19:45:03.883140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.882616 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5"] Nov 02 19:45:03.921404 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:45:03.921368 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca23981_937a_43ec_8571_065f11cd2175.slice/crio-013d235dd5a570e01c483230f8935fc00d574eb778ae9edb2fc4ac1903c14505 WatchSource:0}: Error finding container 013d235dd5a570e01c483230f8935fc00d574eb778ae9edb2fc4ac1903c14505: Status 404 returned error can't find the container with id 013d235dd5a570e01c483230f8935fc00d574eb778ae9edb2fc4ac1903c14505 Nov 02 19:45:03.924688 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.924653 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:45:03.924779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.924737 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:45:03.924827 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:03.924812 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:45:04.844159 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.844107 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" event={"ID":"fca23981-937a-43ec-8571-065f11cd2175","Type":"ContainerStarted","Data":"523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7"} Nov 02 19:45:04.844737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.844199 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" event={"ID":"fca23981-937a-43ec-8571-065f11cd2175","Type":"ContainerStarted","Data":"013d235dd5a570e01c483230f8935fc00d574eb778ae9edb2fc4ac1903c14505"} Nov 02 19:45:04.848065 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.848031 4966 generic.go:356] "Generic (PLEG): container finished" podID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerID="3fe4850c4dade713c42793c0924041ec804cace6b392341d53ae9b386a0349c5" exitCode=60 Nov 02 19:45:04.848308 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.848273 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerDied","Data":"3fe4850c4dade713c42793c0924041ec804cace6b392341d53ae9b386a0349c5"} Nov 02 19:45:04.848391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.848352 4966 scope.go:122] "RemoveContainer" containerID="90aef27334a8ef03a74bbb5fff550d4390fb984f0b90fc422e867767eaaac4f4" Nov 02 19:45:04.853214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.852230 4966 scope.go:122] "RemoveContainer" containerID="3fe4850c4dade713c42793c0924041ec804cace6b392341d53ae9b386a0349c5" Nov 02 19:45:04.853214 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:04.852598 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894145-5gt77_odh-ai-gateway-infra(e1eb5984-46ee-450f-a991-76f890b6fc17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" Nov 02 19:45:04.876137 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.876100 4966 generic.go:356] "Generic (PLEG): container finished" podID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerID="82bc20d3d61075ff7a10dfb0bd700c8cfe2dc4d687779104522abef8082a0c06" exitCode=60 Nov 02 19:45:04.876474 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.876448 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" event={"ID":"f194e2b3-0861-48f0-869e-c7e76883d9ef","Type":"ContainerDied","Data":"82bc20d3d61075ff7a10dfb0bd700c8cfe2dc4d687779104522abef8082a0c06"} Nov 02 19:45:04.877134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.877114 4966 scope.go:122] "RemoveContainer" containerID="82bc20d3d61075ff7a10dfb0bd700c8cfe2dc4d687779104522abef8082a0c06" Nov 02 19:45:04.877579 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:04.877555 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt_odh-ai-gateway-infra(f194e2b3-0861-48f0-869e-c7e76883d9ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" Nov 02 19:45:04.885708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.884753 4966 generic.go:356] "Generic (PLEG): container finished" podID="18ac8ee9-251d-4297-88de-ff5009bef651" containerID="b975feea70aaf9eea2056e2f24319f1398bd6ac797bcc49fb6abb0360e3e13cd" exitCode=60 Nov 02 19:45:04.894161 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.894127 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerDied","Data":"b975feea70aaf9eea2056e2f24319f1398bd6ac797bcc49fb6abb0360e3e13cd"} Nov 02 19:45:04.894819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.894798 4966 scope.go:122] "RemoveContainer" containerID="b975feea70aaf9eea2056e2f24319f1398bd6ac797bcc49fb6abb0360e3e13cd" Nov 02 19:45:04.895274 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:04.895247 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp_odh-ai-gateway-infra(18ac8ee9-251d-4297-88de-ff5009bef651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" Nov 02 19:45:04.904705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.904636 4966 generic.go:356] "Generic (PLEG): container finished" podID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerID="6198092bea8b1fa1d503df41f51dc21349e8c9f75c5fdf84d9cb165d6b5bb82a" exitCode=60 Nov 02 19:45:04.904799 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.904738 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerDied","Data":"6198092bea8b1fa1d503df41f51dc21349e8c9f75c5fdf84d9cb165d6b5bb82a"} Nov 02 19:45:04.905355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.905323 4966 scope.go:122] "RemoveContainer" containerID="6198092bea8b1fa1d503df41f51dc21349e8c9f75c5fdf84d9cb165d6b5bb82a" Nov 02 19:45:04.905669 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:04.905643 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2_odh-ai-gateway-infra(625c5b37-0ad8-4c56-9e72-04a8c5bfb457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" Nov 02 19:45:04.905879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.905860 4966 scope.go:122] "RemoveContainer" containerID="ffcde33b91eb7d6276f8601f1f0e96882577b5c912e19df6df1f56e377c04934" Nov 02 19:45:04.908313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.908278 4966 generic.go:356] "Generic (PLEG): container finished" podID="194b6c77-9946-40b4-b587-9ff7502dc099" containerID="2949cba05512acb6d17e16e211b7845361519cbf6dcc279292180750f65d7094" exitCode=60 Nov 02 19:45:04.908418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.908331 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" event={"ID":"194b6c77-9946-40b4-b587-9ff7502dc099","Type":"ContainerDied","Data":"2949cba05512acb6d17e16e211b7845361519cbf6dcc279292180750f65d7094"} Nov 02 19:45:04.908653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.908635 4966 scope.go:122] "RemoveContainer" containerID="2949cba05512acb6d17e16e211b7845361519cbf6dcc279292180750f65d7094" Nov 02 19:45:04.908900 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:04.908881 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42_odh-ai-gateway-infra(194b6c77-9946-40b4-b587-9ff7502dc099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" Nov 02 19:45:04.921954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.921934 4966 scope.go:122] "RemoveContainer" containerID="0b712f823cff7595355968c7cbcd8046dc7b1154299c451e6b61493f9c758b78" Nov 02 19:45:04.970437 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.969181 4966 scope.go:122] "RemoveContainer" containerID="43b865c840a93405df4f596b7256ebf701f71fde9b7af0bd5ee4ddf93c2f73dc" Nov 02 19:45:04.974604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.973478 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podStartSLOduration=1.973458425 podStartE2EDuration="1.973458425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:04.969977581 +0000 UTC m=+1010.918633421" watchObservedRunningTime="2026-11-02 19:45:04.973458425 +0000 UTC m=+1010.922114265" Nov 02 19:45:04.992761 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:04.992718 4966 scope.go:122] "RemoveContainer" containerID="ed8ead9e726034796ef3d560422a2090fd0ba5381c05c688bef69246c368a90f" Nov 02 19:45:05.449326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:05.447524 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:05.449326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:05.447653 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 19:45:05.449326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:05.447698 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 19:45:05.985715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:05.985685 4966 scope.go:122] "RemoveContainer" containerID="3fe4850c4dade713c42793c0924041ec804cace6b392341d53ae9b386a0349c5" Nov 02 19:45:05.986605 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:05.986576 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894145-5gt77_odh-ai-gateway-infra(e1eb5984-46ee-450f-a991-76f890b6fc17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" Nov 02 19:45:06.052634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:06.029261 4966 scope.go:122] "RemoveContainer" containerID="82bc20d3d61075ff7a10dfb0bd700c8cfe2dc4d687779104522abef8082a0c06" Nov 02 19:45:06.052634 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:06.029619 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt_odh-ai-gateway-infra(f194e2b3-0861-48f0-869e-c7e76883d9ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" Nov 02 19:45:06.052634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:06.033123 4966 scope.go:122] "RemoveContainer" containerID="b975feea70aaf9eea2056e2f24319f1398bd6ac797bcc49fb6abb0360e3e13cd" Nov 02 19:45:06.052634 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:06.038465 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp_odh-ai-gateway-infra(18ac8ee9-251d-4297-88de-ff5009bef651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" Nov 02 19:45:06.052634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:06.049432 4966 scope.go:122] "RemoveContainer" containerID="6198092bea8b1fa1d503df41f51dc21349e8c9f75c5fdf84d9cb165d6b5bb82a" Nov 02 19:45:06.052634 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:06.049740 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2_odh-ai-gateway-infra(625c5b37-0ad8-4c56-9e72-04a8c5bfb457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" Nov 02 19:45:06.079323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:06.061507 4966 scope.go:122] "RemoveContainer" containerID="2949cba05512acb6d17e16e211b7845361519cbf6dcc279292180750f65d7094" Nov 02 19:45:06.079323 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:06.061765 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42_odh-ai-gateway-infra(194b6c77-9946-40b4-b587-9ff7502dc099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" Nov 02 19:45:06.444113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:06.444022 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 19:45:06.444396 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:06.444369 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 19:45:07.459323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:07.444675 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 19:45:07.459323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:07.444738 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 19:45:08.450689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:08.450651 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 19:45:08.450950 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:08.450919 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 19:45:09.443694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:09.443650 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 19:45:09.444236 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:09.444204 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 19:45:10.443909 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:10.443864 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 19:45:10.444331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:10.443934 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 19:45:11.447744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:11.447713 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:11.448198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:11.448187 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:11.448785 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:11.448765 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 19:45:14.754320 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:14.754253 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:45:15.176730 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:15.176652 4966 scope.go:122] "RemoveContainer" containerID="82bc20d3d61075ff7a10dfb0bd700c8cfe2dc4d687779104522abef8082a0c06" Nov 02 19:45:16.183619 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:16.183583 4966 generic.go:356] "Generic (PLEG): container finished" podID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerID="c691ee2ff3fea43d3dd57722d2db6c49aeda26b736631ecf6572489cc1594e5d" exitCode=60 Nov 02 19:45:16.185612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:16.185582 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" event={"ID":"f194e2b3-0861-48f0-869e-c7e76883d9ef","Type":"ContainerDied","Data":"c691ee2ff3fea43d3dd57722d2db6c49aeda26b736631ecf6572489cc1594e5d"} Nov 02 19:45:16.185769 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:16.185653 4966 scope.go:122] "RemoveContainer" containerID="82bc20d3d61075ff7a10dfb0bd700c8cfe2dc4d687779104522abef8082a0c06" Nov 02 19:45:16.186082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:16.186061 4966 scope.go:122] "RemoveContainer" containerID="c691ee2ff3fea43d3dd57722d2db6c49aeda26b736631ecf6572489cc1594e5d" Nov 02 19:45:16.186388 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:16.186366 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt_odh-ai-gateway-infra(f194e2b3-0861-48f0-869e-c7e76883d9ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" Nov 02 19:45:17.176848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.176806 4966 scope.go:122] "RemoveContainer" containerID="3fe4850c4dade713c42793c0924041ec804cace6b392341d53ae9b386a0349c5" Nov 02 19:45:17.177030 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.176968 4966 scope.go:122] "RemoveContainer" containerID="b975feea70aaf9eea2056e2f24319f1398bd6ac797bcc49fb6abb0360e3e13cd" Nov 02 19:45:17.177180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.177160 4966 scope.go:122] "RemoveContainer" containerID="6198092bea8b1fa1d503df41f51dc21349e8c9f75c5fdf84d9cb165d6b5bb82a" Nov 02 19:45:17.306237 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.306195 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt"] Nov 02 19:45:17.478454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.478425 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:17.603125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.603091 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f194e2b3-0861-48f0-869e-c7e76883d9ef-kube-api-access-4tlr8\" (UniqueName: \"kubernetes.io/projected/f194e2b3-0861-48f0-869e-c7e76883d9ef-kube-api-access-4tlr8\") pod \"f194e2b3-0861-48f0-869e-c7e76883d9ef\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " Nov 02 19:45:17.603279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.603233 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f194e2b3-0861-48f0-869e-c7e76883d9ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f194e2b3-0861-48f0-869e-c7e76883d9ef-maas-api-service-ca\") pod \"f194e2b3-0861-48f0-869e-c7e76883d9ef\" (UID: \"f194e2b3-0861-48f0-869e-c7e76883d9ef\") " Nov 02 19:45:17.603612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.603587 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f194e2b3-0861-48f0-869e-c7e76883d9ef-maas-api-service-ca" pod "f194e2b3-0861-48f0-869e-c7e76883d9ef" (UID: "f194e2b3-0861-48f0-869e-c7e76883d9ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:45:17.605624 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.605600 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f194e2b3-0861-48f0-869e-c7e76883d9ef-kube-api-access-4tlr8" pod "f194e2b3-0861-48f0-869e-c7e76883d9ef" (UID: "f194e2b3-0861-48f0-869e-c7e76883d9ef"). InnerVolumeSpecName "kube-api-access-4tlr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:45:17.704896 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.704813 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f194e2b3-0861-48f0-869e-c7e76883d9ef-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:17.704896 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:17.704845 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tlr8\" (UniqueName: \"kubernetes.io/projected/f194e2b3-0861-48f0-869e-c7e76883d9ef-kube-api-access-4tlr8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:18.177281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.176790 4966 scope.go:122] "RemoveContainer" containerID="2949cba05512acb6d17e16e211b7845361519cbf6dcc279292180750f65d7094" Nov 02 19:45:18.223315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.222738 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" event={"ID":"f194e2b3-0861-48f0-869e-c7e76883d9ef","Type":"ContainerDied","Data":"4714d764084dc60f23b6bbc923616f2a059f9392cb25eea82fea79e97271243e"} Nov 02 19:45:18.223315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.222785 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt" Nov 02 19:45:18.223315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.222809 4966 scope.go:122] "RemoveContainer" containerID="c691ee2ff3fea43d3dd57722d2db6c49aeda26b736631ecf6572489cc1594e5d" Nov 02 19:45:18.226054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.226026 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerStarted","Data":"ce9d6a0063ed5105070fda41f8641dab9418d72cdaaa06b27e943cbe27743eb9"} Nov 02 19:45:18.231937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.231912 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerStarted","Data":"45d37f24a9886ff7fd169e88f14d42eeaddadc5412dc711dc43e1c86f97a8660"} Nov 02 19:45:18.234417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.234384 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerStarted","Data":"5cd0fb56f6c26d7171cc536f8326bc3ca5184da32d103aa32698dc1814c8cfa0"} Nov 02 19:45:18.704997 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.704792 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt"] Nov 02 19:45:18.706528 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:18.706502 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5c46af-29894145-56mqt"] Nov 02 19:45:19.249281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.249246 4966 generic.go:356] "Generic (PLEG): container finished" podID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerID="45d37f24a9886ff7fd169e88f14d42eeaddadc5412dc711dc43e1c86f97a8660" exitCode=60 Nov 02 19:45:19.249506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.249353 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerDied","Data":"45d37f24a9886ff7fd169e88f14d42eeaddadc5412dc711dc43e1c86f97a8660"} Nov 02 19:45:19.249506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.249449 4966 scope.go:122] "RemoveContainer" containerID="6198092bea8b1fa1d503df41f51dc21349e8c9f75c5fdf84d9cb165d6b5bb82a" Nov 02 19:45:19.250101 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.249866 4966 scope.go:122] "RemoveContainer" containerID="45d37f24a9886ff7fd169e88f14d42eeaddadc5412dc711dc43e1c86f97a8660" Nov 02 19:45:19.250225 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:19.250187 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2_odh-ai-gateway-infra(625c5b37-0ad8-4c56-9e72-04a8c5bfb457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" Nov 02 19:45:19.252857 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.252827 4966 generic.go:356] "Generic (PLEG): container finished" podID="194b6c77-9946-40b4-b587-9ff7502dc099" containerID="b7518493de42259672ef585ca34d189200725d338d8de155ce49928ae66a24e4" exitCode=60 Nov 02 19:45:19.252972 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.252889 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" event={"ID":"194b6c77-9946-40b4-b587-9ff7502dc099","Type":"ContainerDied","Data":"b7518493de42259672ef585ca34d189200725d338d8de155ce49928ae66a24e4"} Nov 02 19:45:19.253271 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.253253 4966 scope.go:122] "RemoveContainer" containerID="b7518493de42259672ef585ca34d189200725d338d8de155ce49928ae66a24e4" Nov 02 19:45:19.253571 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:19.253548 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42_odh-ai-gateway-infra(194b6c77-9946-40b4-b587-9ff7502dc099)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" Nov 02 19:45:19.255495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.255467 4966 generic.go:356] "Generic (PLEG): container finished" podID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerID="5cd0fb56f6c26d7171cc536f8326bc3ca5184da32d103aa32698dc1814c8cfa0" exitCode=60 Nov 02 19:45:19.255649 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.255633 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerDied","Data":"5cd0fb56f6c26d7171cc536f8326bc3ca5184da32d103aa32698dc1814c8cfa0"} Nov 02 19:45:19.255952 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.255935 4966 scope.go:122] "RemoveContainer" containerID="5cd0fb56f6c26d7171cc536f8326bc3ca5184da32d103aa32698dc1814c8cfa0" Nov 02 19:45:19.256198 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:19.256176 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894145-5gt77_odh-ai-gateway-infra(e1eb5984-46ee-450f-a991-76f890b6fc17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" Nov 02 19:45:19.260682 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.260657 4966 generic.go:356] "Generic (PLEG): container finished" podID="18ac8ee9-251d-4297-88de-ff5009bef651" containerID="ce9d6a0063ed5105070fda41f8641dab9418d72cdaaa06b27e943cbe27743eb9" exitCode=60 Nov 02 19:45:19.260797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.260719 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerDied","Data":"ce9d6a0063ed5105070fda41f8641dab9418d72cdaaa06b27e943cbe27743eb9"} Nov 02 19:45:19.261057 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.261039 4966 scope.go:122] "RemoveContainer" containerID="ce9d6a0063ed5105070fda41f8641dab9418d72cdaaa06b27e943cbe27743eb9" Nov 02 19:45:19.261373 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:19.261292 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp_odh-ai-gateway-infra(18ac8ee9-251d-4297-88de-ff5009bef651)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" Nov 02 19:45:19.268065 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.268024 4966 scope.go:122] "RemoveContainer" containerID="2949cba05512acb6d17e16e211b7845361519cbf6dcc279292180750f65d7094" Nov 02 19:45:19.285372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.285350 4966 scope.go:122] "RemoveContainer" containerID="3fe4850c4dade713c42793c0924041ec804cace6b392341d53ae9b386a0349c5" Nov 02 19:45:19.300401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.300380 4966 scope.go:122] "RemoveContainer" containerID="b975feea70aaf9eea2056e2f24319f1398bd6ac797bcc49fb6abb0360e3e13cd" Nov 02 19:45:19.395819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.395784 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2"] Nov 02 19:45:19.503434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.503270 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp"] Nov 02 19:45:19.616478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:19.616384 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77"] Nov 02 19:45:20.188987 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.188942 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" path="/var/lib/kubelet/pods/f194e2b3-0861-48f0-869e-c7e76883d9ef/volumes" Nov 02 19:45:20.530515 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.530480 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:20.576171 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.576146 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:20.579023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.579006 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:20.658801 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.658771 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1eb5984-46ee-450f-a991-76f890b6fc17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eb5984-46ee-450f-a991-76f890b6fc17-maas-api-service-ca\") pod \"e1eb5984-46ee-450f-a991-76f890b6fc17\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " Nov 02 19:45:20.658962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.658813 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-maas-api-service-ca\") pod \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " Nov 02 19:45:20.658962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.658892 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ac8ee9-251d-4297-88de-ff5009bef651-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ac8ee9-251d-4297-88de-ff5009bef651-maas-api-service-ca\") pod \"18ac8ee9-251d-4297-88de-ff5009bef651\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " Nov 02 19:45:20.658962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.658931 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ac8ee9-251d-4297-88de-ff5009bef651-kube-api-access-8q2ln\" (UniqueName: \"kubernetes.io/projected/18ac8ee9-251d-4297-88de-ff5009bef651-kube-api-access-8q2ln\") pod \"18ac8ee9-251d-4297-88de-ff5009bef651\" (UID: \"18ac8ee9-251d-4297-88de-ff5009bef651\") " Nov 02 19:45:20.659135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.658975 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1eb5984-46ee-450f-a991-76f890b6fc17-kube-api-access-9nj66\" (UniqueName: \"kubernetes.io/projected/e1eb5984-46ee-450f-a991-76f890b6fc17-kube-api-access-9nj66\") pod \"e1eb5984-46ee-450f-a991-76f890b6fc17\" (UID: \"e1eb5984-46ee-450f-a991-76f890b6fc17\") " Nov 02 19:45:20.659135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.659056 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-kube-api-access-8fcp2\" (UniqueName: \"kubernetes.io/projected/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-kube-api-access-8fcp2\") pod \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\" (UID: \"625c5b37-0ad8-4c56-9e72-04a8c5bfb457\") " Nov 02 19:45:20.659229 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.659201 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1eb5984-46ee-450f-a991-76f890b6fc17-maas-api-service-ca" pod "e1eb5984-46ee-450f-a991-76f890b6fc17" (UID: "e1eb5984-46ee-450f-a991-76f890b6fc17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:45:20.659277 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.659232 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-maas-api-service-ca" pod "625c5b37-0ad8-4c56-9e72-04a8c5bfb457" (UID: "625c5b37-0ad8-4c56-9e72-04a8c5bfb457"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:45:20.659370 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.659311 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ac8ee9-251d-4297-88de-ff5009bef651-maas-api-service-ca" pod "18ac8ee9-251d-4297-88de-ff5009bef651" (UID: "18ac8ee9-251d-4297-88de-ff5009bef651"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:45:20.659588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.659571 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eb5984-46ee-450f-a991-76f890b6fc17-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:20.659656 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.659592 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:20.659656 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.659607 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ac8ee9-251d-4297-88de-ff5009bef651-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:20.661170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.661148 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-kube-api-access-8fcp2" pod "625c5b37-0ad8-4c56-9e72-04a8c5bfb457" (UID: "625c5b37-0ad8-4c56-9e72-04a8c5bfb457"). InnerVolumeSpecName "kube-api-access-8fcp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:45:20.661272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.661179 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ac8ee9-251d-4297-88de-ff5009bef651-kube-api-access-8q2ln" pod "18ac8ee9-251d-4297-88de-ff5009bef651" (UID: "18ac8ee9-251d-4297-88de-ff5009bef651"). InnerVolumeSpecName "kube-api-access-8q2ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:45:20.661272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.661180 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1eb5984-46ee-450f-a991-76f890b6fc17-kube-api-access-9nj66" pod "e1eb5984-46ee-450f-a991-76f890b6fc17" (UID: "e1eb5984-46ee-450f-a991-76f890b6fc17"). InnerVolumeSpecName "kube-api-access-9nj66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:45:20.690574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.690549 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42"] Nov 02 19:45:20.761421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.761381 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nj66\" (UniqueName: \"kubernetes.io/projected/e1eb5984-46ee-450f-a991-76f890b6fc17-kube-api-access-9nj66\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:20.761558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.761424 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fcp2\" (UniqueName: \"kubernetes.io/projected/625c5b37-0ad8-4c56-9e72-04a8c5bfb457-kube-api-access-8fcp2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:20.761558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.761442 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2ln\" (UniqueName: \"kubernetes.io/projected/18ac8ee9-251d-4297-88de-ff5009bef651-kube-api-access-8q2ln\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:20.847690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.847658 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:20.963199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.963112 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194b6c77-9946-40b4-b587-9ff7502dc099-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194b6c77-9946-40b4-b587-9ff7502dc099-maas-api-service-ca\") pod \"194b6c77-9946-40b4-b587-9ff7502dc099\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " Nov 02 19:45:20.963199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.963167 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194b6c77-9946-40b4-b587-9ff7502dc099-kube-api-access-hw6gb\" (UniqueName: \"kubernetes.io/projected/194b6c77-9946-40b4-b587-9ff7502dc099-kube-api-access-hw6gb\") pod \"194b6c77-9946-40b4-b587-9ff7502dc099\" (UID: \"194b6c77-9946-40b4-b587-9ff7502dc099\") " Nov 02 19:45:20.963580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.963548 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194b6c77-9946-40b4-b587-9ff7502dc099-maas-api-service-ca" pod "194b6c77-9946-40b4-b587-9ff7502dc099" (UID: "194b6c77-9946-40b4-b587-9ff7502dc099"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:45:20.963766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.963751 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/194b6c77-9946-40b4-b587-9ff7502dc099-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:20.965241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:20.965218 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194b6c77-9946-40b4-b587-9ff7502dc099-kube-api-access-hw6gb" pod "194b6c77-9946-40b4-b587-9ff7502dc099" (UID: "194b6c77-9946-40b4-b587-9ff7502dc099"). InnerVolumeSpecName "kube-api-access-hw6gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:45:21.065692 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.065656 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw6gb\" (UniqueName: \"kubernetes.io/projected/194b6c77-9946-40b4-b587-9ff7502dc099-kube-api-access-hw6gb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:45:21.310321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.309353 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" Nov 02 19:45:21.310321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.309346 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2" event={"ID":"625c5b37-0ad8-4c56-9e72-04a8c5bfb457","Type":"ContainerDied","Data":"df674f0bc3b1cd6e4ea2e2c42fb9dc50dd4ac7dc6e581a60f7be7cf45727fab4"} Nov 02 19:45:21.310321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.309510 4966 scope.go:122] "RemoveContainer" containerID="45d37f24a9886ff7fd169e88f14d42eeaddadc5412dc711dc43e1c86f97a8660" Nov 02 19:45:21.316323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.311380 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" event={"ID":"194b6c77-9946-40b4-b587-9ff7502dc099","Type":"ContainerDied","Data":"2e0365d5ff009f8571277fd579b989888a2b4c5848136c1d0024a90e03a8ce74"} Nov 02 19:45:21.316323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.311387 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42" Nov 02 19:45:21.316323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.313092 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" event={"ID":"e1eb5984-46ee-450f-a991-76f890b6fc17","Type":"ContainerDied","Data":"c61b35511b82e4b8933ee18879216c0496aa8457260b35f1d3f32f1eda42ee88"} Nov 02 19:45:21.316323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.313171 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77" Nov 02 19:45:21.316323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.314926 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" event={"ID":"18ac8ee9-251d-4297-88de-ff5009bef651","Type":"ContainerDied","Data":"fc205962627da6430b62b3e3bffb88a868b123fd9d1f3a4ae959f7637476168b"} Nov 02 19:45:21.316323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.315021 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp" Nov 02 19:45:21.325687 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.325637 4966 scope.go:122] "RemoveContainer" containerID="b7518493de42259672ef585ca34d189200725d338d8de155ce49928ae66a24e4" Nov 02 19:45:21.340465 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.340435 4966 scope.go:122] "RemoveContainer" containerID="5cd0fb56f6c26d7171cc536f8326bc3ca5184da32d103aa32698dc1814c8cfa0" Nov 02 19:45:21.350486 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.350463 4966 scope.go:122] "RemoveContainer" containerID="ce9d6a0063ed5105070fda41f8641dab9418d72cdaaa06b27e943cbe27743eb9" Nov 02 19:45:21.466995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.466959 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2"] Nov 02 19:45:21.473250 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.473220 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894145-6klq2"] Nov 02 19:45:21.572769 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.572680 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp"] Nov 02 19:45:21.577934 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.577901 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e9c144-29894145-9pkkp"] Nov 02 19:45:21.680067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.680032 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77"] Nov 02 19:45:21.684223 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.684191 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894145-5gt77"] Nov 02 19:45:21.782220 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.782170 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42"] Nov 02 19:45:21.786177 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:21.786151 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894145-d6k42"] Nov 02 19:45:22.185815 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:22.185772 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" path="/var/lib/kubelet/pods/18ac8ee9-251d-4297-88de-ff5009bef651/volumes" Nov 02 19:45:22.186374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:22.186356 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" path="/var/lib/kubelet/pods/194b6c77-9946-40b4-b587-9ff7502dc099/volumes" Nov 02 19:45:22.186749 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:22.186734 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" path="/var/lib/kubelet/pods/625c5b37-0ad8-4c56-9e72-04a8c5bfb457/volumes" Nov 02 19:45:22.187404 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:22.187389 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" path="/var/lib/kubelet/pods/e1eb5984-46ee-450f-a991-76f890b6fc17/volumes" Nov 02 19:45:37.974279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974182 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz"] Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974737 4966 cpu_manager.go:414] "containerMap: removing container" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974751 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974761 4966 cpu_manager.go:414] "containerMap: removing container" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974766 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974778 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974783 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974789 4966 cpu_manager.go:414] "containerMap: removing container" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974793 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974801 4966 cpu_manager.go:414] "containerMap: removing container" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974805 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974849 4966 cpu_manager.go:414] "containerMap: removing container" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974854 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974864 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974868 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974879 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974885 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974893 4966 cpu_manager.go:414] "containerMap: removing container" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974898 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:45:37.974894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974905 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974910 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974918 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.974922 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975006 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975013 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975021 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975028 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975034 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975039 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975046 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eb5984-46ee-450f-a991-76f890b6fc17" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975052 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975057 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975064 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975073 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:45:37.975848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.975077 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:45:37.978957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:37.978935 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.033053 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.033019 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz"] Nov 02 19:45:38.074123 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.074081 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e9c144\"/\"distinct-2-w6-e9c144-kserve-self-signed-certs\"" Nov 02 19:45:38.155970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.155926 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-home\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.156242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.156006 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tmp-dir\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.156242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.156176 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tls-certs\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.156242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.156235 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-dshm\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.156443 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.156341 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w4g\" (UniqueName: \"kubernetes.io/projected/5d0587f2-6194-414c-a41a-ac1d9669dc0d-kube-api-access-r6w4g\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.156443 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.156385 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-model-cache\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.257505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.257404 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-home\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.257505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.257481 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tmp-dir\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.257736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.257565 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tls-certs\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.257736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.257620 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-dshm\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.257736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.257649 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w4g\" (UniqueName: \"kubernetes.io/projected/5d0587f2-6194-414c-a41a-ac1d9669dc0d-kube-api-access-r6w4g\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.257736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.257672 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-model-cache\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.258460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.258432 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tmp-dir\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.258593 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.258472 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-home\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.258593 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.258512 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-model-cache\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.261458 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.260879 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-dshm\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.263899 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.263854 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tls-certs\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.315337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.310690 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w4g\" (UniqueName: \"kubernetes.io/projected/5d0587f2-6194-414c-a41a-ac1d9669dc0d-kube-api-access-r6w4g\") pod \"distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.592801 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.592689 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:38.976265 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:45:38.976233 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d0587f2_6194_414c_a41a_ac1d9669dc0d.slice/crio-5e2616c67eb87b61ed2ae0d82885184fc352ed0ba8571457671666b4dce68c06 WatchSource:0}: Error finding container 5e2616c67eb87b61ed2ae0d82885184fc352ed0ba8571457671666b4dce68c06: Status 404 returned error can't find the container with id 5e2616c67eb87b61ed2ae0d82885184fc352ed0ba8571457671666b4dce68c06 Nov 02 19:45:38.978040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:38.977999 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz"] Nov 02 19:45:39.471278 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:39.471239 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" event={"ID":"5d0587f2-6194-414c-a41a-ac1d9669dc0d","Type":"ContainerStarted","Data":"64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2"} Nov 02 19:45:39.471488 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:39.471327 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" event={"ID":"5d0587f2-6194-414c-a41a-ac1d9669dc0d","Type":"ContainerStarted","Data":"5e2616c67eb87b61ed2ae0d82885184fc352ed0ba8571457671666b4dce68c06"} Nov 02 19:45:39.579164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:39.579072 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podStartSLOduration=2.579055994 podStartE2EDuration="2.579055994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:45:39.576408254 +0000 UTC m=+1045.525064093" watchObservedRunningTime="2026-11-02 19:45:39.579055994 +0000 UTC m=+1045.527711834" Nov 02 19:45:39.593440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:39.593406 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:44.752701 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:45:44.752660 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:45:48.592851 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:48.592808 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:48.606839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:48.606809 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:45:49.565290 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:45:49.565256 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:46:14.751606 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:46:14.751565 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:46:19.814867 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.814833 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758"] Nov 02 19:46:19.815361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815346 4966 cpu_manager.go:414] "containerMap: removing container" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:46:19.815418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815362 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:46:19.815418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815378 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:46:19.815418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815384 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:46:19.815418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815401 4966 cpu_manager.go:414] "containerMap: removing container" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:46:19.815418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815405 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ac8ee9-251d-4297-88de-ff5009bef651" containerName="cleanup" Nov 02 19:46:19.815418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815416 4966 cpu_manager.go:414] "containerMap: removing container" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:46:19.815595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815420 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:46:19.815595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815521 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="625c5b37-0ad8-4c56-9e72-04a8c5bfb457" containerName="cleanup" Nov 02 19:46:19.815595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815531 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="194b6c77-9946-40b4-b587-9ff7502dc099" containerName="cleanup" Nov 02 19:46:19.815595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.815543 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f194e2b3-0861-48f0-869e-c7e76883d9ef" containerName="cleanup" Nov 02 19:46:19.820797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.820777 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.869575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.869529 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e9c144-openshift-default-dockercfg-7slsb\"" Nov 02 19:46:19.922667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.922632 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758"] Nov 02 19:46:19.936338 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936273 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936394 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936510 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936486 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936562 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936599 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936677 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936757 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvhbv\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-kube-api-access-pvhbv\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936826 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936812 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:19.936866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:19.936852 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.037892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.037850 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.037892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.037897 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038136 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.037992 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038136 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038029 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038136 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038046 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038136 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038113 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038143 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvhbv\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-kube-api-access-pvhbv\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038346 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038364 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038409 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038428 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038497 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.038794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.038775 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.039023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.039000 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.040738 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.040711 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.040948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.040932 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.091725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.091655 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.091879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.091854 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvhbv\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-kube-api-access-pvhbv\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.135195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.135155 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:20.579735 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:46:20.579702 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b367fbd_ccfe_4cc6_8771_4ee083ae0422.slice/crio-6719392bc388f4df008332169e093fdb83f87e9719e66e1468d91281d7fecec3 WatchSource:0}: Error finding container 6719392bc388f4df008332169e093fdb83f87e9719e66e1468d91281d7fecec3: Status 404 returned error can't find the container with id 6719392bc388f4df008332169e093fdb83f87e9719e66e1468d91281d7fecec3 Nov 02 19:46:20.579864 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.579799 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758"] Nov 02 19:46:20.582693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.582659 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:46:20.582774 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.582738 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:46:20.582813 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.582779 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:46:20.819317 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.818658 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" event={"ID":"0b367fbd-ccfe-4cc6-8771-4ee083ae0422","Type":"ContainerStarted","Data":"3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae"} Nov 02 19:46:20.819317 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.818721 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" event={"ID":"0b367fbd-ccfe-4cc6-8771-4ee083ae0422","Type":"ContainerStarted","Data":"6719392bc388f4df008332169e093fdb83f87e9719e66e1468d91281d7fecec3"} Nov 02 19:46:20.936105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:20.935992 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podStartSLOduration=1.935973147 podStartE2EDuration="1.935973147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:46:20.929255054 +0000 UTC m=+1086.877910893" watchObservedRunningTime="2026-11-02 19:46:20.935973147 +0000 UTC m=+1086.884628989" Nov 02 19:46:21.135572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.135529 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:21.137343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.137288 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 19:46:21.137509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.137367 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 19:46:21.282509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.282469 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd"] Nov 02 19:46:21.287604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.287581 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.353724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.353688 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-43ceaa\"/\"distinct-2-w0-43ceaa-kserve-self-signed-certs\"" Nov 02 19:46:21.354763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.354731 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-home\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.354949 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.354930 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e604c-6be5-487a-a55a-669b6c22f9aa-tls-certs\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.355114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.355097 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-dshm\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.355278 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.355260 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsz6q\" (UniqueName: \"kubernetes.io/projected/378e604c-6be5-487a-a55a-669b6c22f9aa-kube-api-access-tsz6q\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.355465 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.355448 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-tmp-dir\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.366320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.355632 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-model-cache\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.381815 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.381757 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd"] Nov 02 19:46:21.483082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483040 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-tmp-dir\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.483287 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483185 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-model-cache\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.483670 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483647 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-home\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.483732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483684 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e604c-6be5-487a-a55a-669b6c22f9aa-tls-certs\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.483732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483689 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-model-cache\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.483840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483788 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-dshm\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.483893 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483867 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsz6q\" (UniqueName: \"kubernetes.io/projected/378e604c-6be5-487a-a55a-669b6c22f9aa-kube-api-access-tsz6q\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.483955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.483933 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-home\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.485344 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.484290 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-tmp-dir\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.486234 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.486211 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-dshm\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.488092 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.488074 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e604c-6be5-487a-a55a-669b6c22f9aa-tls-certs\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.541367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.540977 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsz6q\" (UniqueName: \"kubernetes.io/projected/378e604c-6be5-487a-a55a-669b6c22f9aa-kube-api-access-tsz6q\") pod \"distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:21.603059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:21.603020 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:22.090852 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:46:22.090806 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod378e604c_6be5_487a_a55a_669b6c22f9aa.slice/crio-71b8ee9de05543cba8a7538ee87225d7ee38acccf78ce5d6477e97f18bf29669 WatchSource:0}: Error finding container 71b8ee9de05543cba8a7538ee87225d7ee38acccf78ce5d6477e97f18bf29669: Status 404 returned error can't find the container with id 71b8ee9de05543cba8a7538ee87225d7ee38acccf78ce5d6477e97f18bf29669 Nov 02 19:46:22.094805 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.094773 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd"] Nov 02 19:46:22.136341 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.136310 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 19:46:22.136468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.136365 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 19:46:22.569056 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.569020 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2"] Nov 02 19:46:22.573265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.573244 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.616209 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.610480 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2"] Nov 02 19:46:22.621887 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.621861 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:46:22.622037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.621930 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:46:22.622256 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.622229 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:46:22.622420 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.622351 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:46:22.716713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.716678 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.716906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.716722 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxf2x\" (UniqueName: \"kubernetes.io/projected/aa4865ee-8920-4465-9cb6-341174df580d-kube-api-access-xxf2x\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.716906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.716827 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.817788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.817750 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.817788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.817789 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxf2x\" (UniqueName: \"kubernetes.io/projected/aa4865ee-8920-4465-9cb6-341174df580d-kube-api-access-xxf2x\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.818046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.817826 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.818666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.818635 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.820357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.820291 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.837094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.837057 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" event={"ID":"378e604c-6be5-487a-a55a-669b6c22f9aa","Type":"ContainerStarted","Data":"499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4"} Nov 02 19:46:22.837259 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.837111 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" event={"ID":"378e604c-6be5-487a-a55a-669b6c22f9aa","Type":"ContainerStarted","Data":"71b8ee9de05543cba8a7538ee87225d7ee38acccf78ce5d6477e97f18bf29669"} Nov 02 19:46:22.871281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.871234 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxf2x\" (UniqueName: \"kubernetes.io/projected/aa4865ee-8920-4465-9cb6-341174df580d-kube-api-access-xxf2x\") pod \"maas-api-revoke-sub-effce9fb-rjlz2\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.888948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.888915 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:22.972762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:22.970070 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podStartSLOduration=1.970048367 podStartE2EDuration="1.970048367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:46:22.955472935 +0000 UTC m=+1088.904128775" watchObservedRunningTime="2026-11-02 19:46:22.970048367 +0000 UTC m=+1088.918704210" Nov 02 19:46:23.136108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:23.136026 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 19:46:23.136108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:23.136085 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 19:46:23.282988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:23.282946 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2"] Nov 02 19:46:23.286731 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:46:23.286680 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4865ee_8920_4465_9cb6_341174df580d.slice/crio-4b67a80fbe82fee7eba884129a37b553054f616b443023a58f3f65c4ed005890 WatchSource:0}: Error finding container 4b67a80fbe82fee7eba884129a37b553054f616b443023a58f3f65c4ed005890: Status 404 returned error can't find the container with id 4b67a80fbe82fee7eba884129a37b553054f616b443023a58f3f65c4ed005890 Nov 02 19:46:23.603110 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:23.603068 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:23.870807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:23.870713 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" event={"ID":"aa4865ee-8920-4465-9cb6-341174df580d","Type":"ContainerStarted","Data":"43ae5e6338cebd2d500754d2f359658cab663a4d31e772d9dd375e7b85c096b8"} Nov 02 19:46:23.870807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:23.870764 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" event={"ID":"aa4865ee-8920-4465-9cb6-341174df580d","Type":"ContainerStarted","Data":"4b67a80fbe82fee7eba884129a37b553054f616b443023a58f3f65c4ed005890"} Nov 02 19:46:23.980845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:23.980788 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" podStartSLOduration=1.980774297 podStartE2EDuration="1.980774297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:46:23.976373825 +0000 UTC m=+1089.925029663" watchObservedRunningTime="2026-11-02 19:46:23.980774297 +0000 UTC m=+1089.929430173" Nov 02 19:46:24.136216 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:24.136119 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 19:46:24.136216 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:24.136187 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 19:46:24.883388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:24.883281 4966 generic.go:356] "Generic (PLEG): container finished" podID="aa4865ee-8920-4465-9cb6-341174df580d" containerID="43ae5e6338cebd2d500754d2f359658cab663a4d31e772d9dd375e7b85c096b8" exitCode=0 Nov 02 19:46:24.883388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:24.883372 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" event={"ID":"aa4865ee-8920-4465-9cb6-341174df580d","Type":"ContainerDied","Data":"43ae5e6338cebd2d500754d2f359658cab663a4d31e772d9dd375e7b85c096b8"} Nov 02 19:46:25.136540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:25.136459 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 19:46:25.136540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:25.136511 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 19:46:26.097045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.097020 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:26.136004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.135970 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 19:46:26.136183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.136028 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 19:46:26.183701 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.183603 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-openshift-ca\") pod \"aa4865ee-8920-4465-9cb6-341174df580d\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " Nov 02 19:46:26.184140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.183770 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-xks-ca\") pod \"aa4865ee-8920-4465-9cb6-341174df580d\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " Nov 02 19:46:26.184140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.183834 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4865ee-8920-4465-9cb6-341174df580d-kube-api-access-xxf2x\" (UniqueName: \"kubernetes.io/projected/aa4865ee-8920-4465-9cb6-341174df580d-kube-api-access-xxf2x\") pod \"aa4865ee-8920-4465-9cb6-341174df580d\" (UID: \"aa4865ee-8920-4465-9cb6-341174df580d\") " Nov 02 19:46:26.184140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.184073 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-openshift-ca" pod "aa4865ee-8920-4465-9cb6-341174df580d" (UID: "aa4865ee-8920-4465-9cb6-341174df580d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:46:26.184322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.184288 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:46:26.186048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.186011 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-xks-ca" pod "aa4865ee-8920-4465-9cb6-341174df580d" (UID: "aa4865ee-8920-4465-9cb6-341174df580d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:46:26.186444 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.186417 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4865ee-8920-4465-9cb6-341174df580d-kube-api-access-xxf2x" pod "aa4865ee-8920-4465-9cb6-341174df580d" (UID: "aa4865ee-8920-4465-9cb6-341174df580d"). InnerVolumeSpecName "kube-api-access-xxf2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:46:26.293317 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.286419 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4865ee-8920-4465-9cb6-341174df580d-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:46:26.293317 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.286457 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxf2x\" (UniqueName: \"kubernetes.io/projected/aa4865ee-8920-4465-9cb6-341174df580d-kube-api-access-xxf2x\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:46:26.908019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.907967 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" Nov 02 19:46:26.908233 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.908030 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2" event={"ID":"aa4865ee-8920-4465-9cb6-341174df580d","Type":"ContainerDied","Data":"4b67a80fbe82fee7eba884129a37b553054f616b443023a58f3f65c4ed005890"} Nov 02 19:46:26.908233 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:26.908128 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b67a80fbe82fee7eba884129a37b553054f616b443023a58f3f65c4ed005890" Nov 02 19:46:27.135536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:27.135505 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 19:46:27.135749 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:27.135635 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 19:46:28.140469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:28.140436 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:28.140977 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:28.140792 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:28.141691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:28.141660 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 19:46:31.604069 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:31.604032 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:31.616169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:31.616145 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:31.957392 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:46:31.957263 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:46:44.752855 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:46:44.752815 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:47:02.423655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.423571 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w"] Nov 02 19:47:02.424124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.424091 4966 cpu_manager.go:414] "containerMap: removing container" podUID="aa4865ee-8920-4465-9cb6-341174df580d" containerName="revoke-keys" Nov 02 19:47:02.424124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.424103 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4865ee-8920-4465-9cb6-341174df580d" containerName="revoke-keys" Nov 02 19:47:02.424243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.424231 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4865ee-8920-4465-9cb6-341174df580d" containerName="revoke-keys" Nov 02 19:47:02.427729 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.427711 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.480556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.480521 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:47:02.534354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.531586 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:47:02.534354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.531586 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:47:02.534354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.531594 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:47:02.534354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.532045 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.534354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.532096 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6q9\" (UniqueName: \"kubernetes.io/projected/a9c04566-fcff-481f-945f-f2329c8d3897-kube-api-access-fr6q9\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.534354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.532125 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.538556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.538525 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w"] Nov 02 19:47:02.632905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.632869 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.633127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.633006 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.633127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.633050 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6q9\" (UniqueName: \"kubernetes.io/projected/a9c04566-fcff-481f-945f-f2329c8d3897-kube-api-access-fr6q9\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.633799 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.633771 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.635518 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.635497 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.688458 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.688384 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6q9\" (UniqueName: \"kubernetes.io/projected/a9c04566-fcff-481f-945f-f2329c8d3897-kube-api-access-fr6q9\") pod \"maas-api-revoke-sub-4512810e-nng9w\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:02.741951 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:02.741917 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:03.115432 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:03.115400 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c04566_fcff_481f_945f_f2329c8d3897.slice/crio-c47b4c1f0275e1a977a1c235b3b491335517f4794ca5be253f02aa8e84a852a3 WatchSource:0}: Error finding container c47b4c1f0275e1a977a1c235b3b491335517f4794ca5be253f02aa8e84a852a3: Status 404 returned error can't find the container with id c47b4c1f0275e1a977a1c235b3b491335517f4794ca5be253f02aa8e84a852a3 Nov 02 19:47:03.117631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:03.117604 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w"] Nov 02 19:47:03.223034 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:03.222998 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" event={"ID":"a9c04566-fcff-481f-945f-f2329c8d3897","Type":"ContainerStarted","Data":"c47b4c1f0275e1a977a1c235b3b491335517f4794ca5be253f02aa8e84a852a3"} Nov 02 19:47:04.232851 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:04.232817 4966 generic.go:356] "Generic (PLEG): container finished" podID="a9c04566-fcff-481f-945f-f2329c8d3897" containerID="c74f5485b5b6d2a9ad4eda8e37381c265a483b03ab63eff6acf8ad49511adf52" exitCode=0 Nov 02 19:47:04.233249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:04.232907 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" event={"ID":"a9c04566-fcff-481f-945f-f2329c8d3897","Type":"ContainerDied","Data":"c74f5485b5b6d2a9ad4eda8e37381c265a483b03ab63eff6acf8ad49511adf52"} Nov 02 19:47:04.917626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:04.917582 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2"] Nov 02 19:47:04.921927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:04.921902 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:04.981562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:04.981525 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2"] Nov 02 19:47:05.061743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.061705 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a4e90c-b1b7-4414-9d14-9355e04a0511-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.061933 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.061769 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.061933 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.061896 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.062021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.061984 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.062076 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.062032 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.062137 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.062123 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.062174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.062156 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqx2t\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-kube-api-access-kqx2t\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.062246 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.062235 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.062310 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.062287 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167202 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167271 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqx2t\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-kube-api-access-kqx2t\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167354 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167432 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167500 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a4e90c-b1b7-4414-9d14-9355e04a0511-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167543 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167602 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167678 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.167731 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.169366 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a4e90c-b1b7-4414-9d14-9355e04a0511-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.170282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.169936 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.174828 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.174798 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.174979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.174878 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.175052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.175035 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.176985 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.176953 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.177179 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.177157 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.225494 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.225457 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.226450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.226423 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqx2t\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-kube-api-access-kqx2t\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.237405 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.237379 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:05.438511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.438475 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:05.574316 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.574266 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-xks-ca\") pod \"a9c04566-fcff-481f-945f-f2329c8d3897\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " Nov 02 19:47:05.574502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.574434 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c04566-fcff-481f-945f-f2329c8d3897-kube-api-access-fr6q9\" (UniqueName: \"kubernetes.io/projected/a9c04566-fcff-481f-945f-f2329c8d3897-kube-api-access-fr6q9\") pod \"a9c04566-fcff-481f-945f-f2329c8d3897\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " Nov 02 19:47:05.574502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.574491 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-openshift-ca\") pod \"a9c04566-fcff-481f-945f-f2329c8d3897\" (UID: \"a9c04566-fcff-481f-945f-f2329c8d3897\") " Nov 02 19:47:05.575038 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.575011 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-openshift-ca" pod "a9c04566-fcff-481f-945f-f2329c8d3897" (UID: "a9c04566-fcff-481f-945f-f2329c8d3897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:47:05.576716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.576688 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c04566-fcff-481f-945f-f2329c8d3897-kube-api-access-fr6q9" pod "a9c04566-fcff-481f-945f-f2329c8d3897" (UID: "a9c04566-fcff-481f-945f-f2329c8d3897"). InnerVolumeSpecName "kube-api-access-fr6q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:47:05.576839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.576820 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-xks-ca" pod "a9c04566-fcff-481f-945f-f2329c8d3897" (UID: "a9c04566-fcff-481f-945f-f2329c8d3897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:47:05.675892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.675856 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:05.675892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.675887 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6q9\" (UniqueName: \"kubernetes.io/projected/a9c04566-fcff-481f-945f-f2329c8d3897-kube-api-access-fr6q9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:05.675892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.675897 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c04566-fcff-481f-945f-f2329c8d3897-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:05.701115 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:05.701075 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a4e90c_b1b7_4414_9d14_9355e04a0511.slice/crio-7a8a0f46dc07e7dfec45ff6c923eff79200db19669b639622a6bf53258cf3d7f WatchSource:0}: Error finding container 7a8a0f46dc07e7dfec45ff6c923eff79200db19669b639622a6bf53258cf3d7f: Status 404 returned error can't find the container with id 7a8a0f46dc07e7dfec45ff6c923eff79200db19669b639622a6bf53258cf3d7f Nov 02 19:47:05.701532 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.701507 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2"] Nov 02 19:47:05.703633 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.703597 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:05.703744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.703676 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:05.703744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:05.703721 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:06.254412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:06.254379 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" Nov 02 19:47:06.254894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:06.254409 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w" event={"ID":"a9c04566-fcff-481f-945f-f2329c8d3897","Type":"ContainerDied","Data":"c47b4c1f0275e1a977a1c235b3b491335517f4794ca5be253f02aa8e84a852a3"} Nov 02 19:47:06.254894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:06.254459 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c47b4c1f0275e1a977a1c235b3b491335517f4794ca5be253f02aa8e84a852a3" Nov 02 19:47:06.256922 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:06.256890 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" event={"ID":"01a4e90c-b1b7-4414-9d14-9355e04a0511","Type":"ContainerStarted","Data":"8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106"} Nov 02 19:47:06.257186 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:06.257164 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" event={"ID":"01a4e90c-b1b7-4414-9d14-9355e04a0511","Type":"ContainerStarted","Data":"7a8a0f46dc07e7dfec45ff6c923eff79200db19669b639622a6bf53258cf3d7f"} Nov 02 19:47:06.373259 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:06.373134 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podStartSLOduration=2.373113743 podStartE2EDuration="2.373113743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:06.370887489 +0000 UTC m=+1132.319543330" watchObservedRunningTime="2026-11-02 19:47:06.373113743 +0000 UTC m=+1132.321769582" Nov 02 19:47:07.254322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:07.241598 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:07.255461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:07.255424 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 19:47:07.256034 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:07.256001 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 19:47:08.242325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:08.241653 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 19:47:08.242325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:08.241714 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 19:47:08.972323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:08.968166 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6"] Nov 02 19:47:08.978318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:08.976199 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a9c04566-fcff-481f-945f-f2329c8d3897" containerName="revoke-keys" Nov 02 19:47:08.978318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:08.976233 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c04566-fcff-481f-945f-f2329c8d3897" containerName="revoke-keys" Nov 02 19:47:08.978318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:08.976461 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c04566-fcff-481f-945f-f2329c8d3897" containerName="revoke-keys" Nov 02 19:47:08.981244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:08.981218 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.011240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.011202 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6"] Nov 02 19:47:09.017367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.017340 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-model-cache\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.017507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.017397 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-dshm\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.017507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.017491 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tmp-dir\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.017621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.017585 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tls-certs\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.017753 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.017731 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grb4q\" (UniqueName: \"kubernetes.io/projected/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-kube-api-access-grb4q\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.017818 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.017801 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-home\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.057336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.056638 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26cfbe\"/\"distinct-2-w1-26cfbe-kserve-self-signed-certs\"" Nov 02 19:47:09.121429 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.121388 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-model-cache\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.121636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.121499 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-dshm\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.121636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.121548 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tmp-dir\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.121765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.121646 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tls-certs\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.121765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.121729 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb4q\" (UniqueName: \"kubernetes.io/projected/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-kube-api-access-grb4q\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.121888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.121809 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-home\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.122035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.122015 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-model-cache\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.122130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.122066 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tmp-dir\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.122691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.122668 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-home\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.124170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.124147 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-dshm\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.125963 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.125941 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tls-certs\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.188609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.188532 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grb4q\" (UniqueName: \"kubernetes.io/projected/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-kube-api-access-grb4q\") pod \"distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.238158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.238073 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 19:47:09.238425 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.238397 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 19:47:09.302549 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.302516 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:09.811355 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:09.811320 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd80b1bff_24ef_4e5c_bd1e_4b57a1654eda.slice/crio-19821b9d74dae6395c87615d4b4f71a648ebfa642a2352f37d22558b4fe967b9 WatchSource:0}: Error finding container 19821b9d74dae6395c87615d4b4f71a648ebfa642a2352f37d22558b4fe967b9: Status 404 returned error can't find the container with id 19821b9d74dae6395c87615d4b4f71a648ebfa642a2352f37d22558b4fe967b9 Nov 02 19:47:09.812043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:09.812015 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6"] Nov 02 19:47:10.251322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:10.242616 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 19:47:10.251322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:10.242677 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 19:47:10.313471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:10.313394 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" event={"ID":"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda","Type":"ContainerStarted","Data":"526d625524b07af010225b37a324e743092016340a94186d6ef3dadd44842a20"} Nov 02 19:47:10.313471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:10.313434 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" event={"ID":"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda","Type":"ContainerStarted","Data":"19821b9d74dae6395c87615d4b4f71a648ebfa642a2352f37d22558b4fe967b9"} Nov 02 19:47:10.461324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:10.459698 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podStartSLOduration=2.459678019 podStartE2EDuration="2.459678019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:10.436244528 +0000 UTC m=+1136.384900367" watchObservedRunningTime="2026-11-02 19:47:10.459678019 +0000 UTC m=+1136.408333858" Nov 02 19:47:11.260321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:11.247323 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 19:47:11.260321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:11.247383 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 19:47:11.302625 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:11.302595 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:12.238181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:12.238146 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 19:47:12.238405 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:12.238208 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 19:47:13.381326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:13.377896 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:13.381326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:13.379556 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:13.385320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:13.381935 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:47:14.758319 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:47:14.758270 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:47:15.059077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:15.058975 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz"] Nov 02 19:47:15.103174 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:15.103138 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4d65c3-8tjxz"] Nov 02 19:47:16.208957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:16.208924 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72e0614-b5ff-445f-9f5b-37c18577ce19" path="/var/lib/kubelet/pods/f72e0614-b5ff-445f-9f5b-37c18577ce19/volumes" Nov 02 19:47:17.995615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:17.995576 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t"] Nov 02 19:47:18.001084 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.001064 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.056081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.056045 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t"] Nov 02 19:47:18.083481 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.083445 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.083658 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.083630 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrfwr\" (UniqueName: \"kubernetes.io/projected/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-kube-api-access-nrfwr\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.083779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.083758 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.095272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.095222 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:47:18.095272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.095232 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:47:18.095586 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.095567 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:47:18.096527 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.096495 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:47:18.184751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.184717 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.184950 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.184931 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.185045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.185028 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrfwr\" (UniqueName: \"kubernetes.io/projected/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-kube-api-access-nrfwr\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.185672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.185650 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.187540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.187517 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.238386 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.238356 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrfwr\" (UniqueName: \"kubernetes.io/projected/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-kube-api-access-nrfwr\") pod \"maas-api-revoke-sub-e12e5877-qzc5t\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.314551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.314456 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:18.688286 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:18.688253 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t"] Nov 02 19:47:18.688641 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:18.688615 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0e97b3_1653_4b94_a4cc_ea5e10e70668.slice/crio-2d19ab65fb0ad95e99f8c427f5e51d0338813f94d057c8a9a95ecef162bf1acd WatchSource:0}: Error finding container 2d19ab65fb0ad95e99f8c427f5e51d0338813f94d057c8a9a95ecef162bf1acd: Status 404 returned error can't find the container with id 2d19ab65fb0ad95e99f8c427f5e51d0338813f94d057c8a9a95ecef162bf1acd Nov 02 19:47:19.302995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:19.302961 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:19.316416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:19.316384 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:19.459544 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:19.459501 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" event={"ID":"8a0e97b3-1653-4b94-a4cc-ea5e10e70668","Type":"ContainerStarted","Data":"e10f31932b251766c8f1115ae9155f8b5816ef5dfef77ee4c8bf3b75be207874"} Nov 02 19:47:19.459736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:19.459586 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" event={"ID":"8a0e97b3-1653-4b94-a4cc-ea5e10e70668","Type":"ContainerStarted","Data":"2d19ab65fb0ad95e99f8c427f5e51d0338813f94d057c8a9a95ecef162bf1acd"} Nov 02 19:47:19.473774 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:19.473743 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:47:19.569591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:19.569497 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" podStartSLOduration=2.5694831369999998 podStartE2EDuration="2.569483137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:19.564863503 +0000 UTC m=+1145.513519342" watchObservedRunningTime="2026-11-02 19:47:19.569483137 +0000 UTC m=+1145.518138976" Nov 02 19:47:20.048109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:20.048075 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl"] Nov 02 19:47:20.054684 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:20.054653 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcfaaf09-6fphl"] Nov 02 19:47:20.184352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:20.184311 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24276cd-e447-4f78-a8e7-c9c4f09efb9b" path="/var/lib/kubelet/pods/c24276cd-e447-4f78-a8e7-c9c4f09efb9b/volumes" Nov 02 19:47:20.469809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:20.469764 4966 generic.go:356] "Generic (PLEG): container finished" podID="8a0e97b3-1653-4b94-a4cc-ea5e10e70668" containerID="e10f31932b251766c8f1115ae9155f8b5816ef5dfef77ee4c8bf3b75be207874" exitCode=0 Nov 02 19:47:20.470257 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:20.469835 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" event={"ID":"8a0e97b3-1653-4b94-a4cc-ea5e10e70668","Type":"ContainerDied","Data":"e10f31932b251766c8f1115ae9155f8b5816ef5dfef77ee4c8bf3b75be207874"} Nov 02 19:47:21.650440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.650414 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:21.727856 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.727814 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-openshift-ca\") pod \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " Nov 02 19:47:21.728008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.727949 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-xks-ca\") pod \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " Nov 02 19:47:21.728008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.727993 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-kube-api-access-nrfwr\" (UniqueName: \"kubernetes.io/projected/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-kube-api-access-nrfwr\") pod \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\" (UID: \"8a0e97b3-1653-4b94-a4cc-ea5e10e70668\") " Nov 02 19:47:21.728206 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.728179 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-openshift-ca" pod "8a0e97b3-1653-4b94-a4cc-ea5e10e70668" (UID: "8a0e97b3-1653-4b94-a4cc-ea5e10e70668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:47:21.728443 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.728424 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:21.730185 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.730158 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-xks-ca" pod "8a0e97b3-1653-4b94-a4cc-ea5e10e70668" (UID: "8a0e97b3-1653-4b94-a4cc-ea5e10e70668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:47:21.730343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.730184 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-kube-api-access-nrfwr" pod "8a0e97b3-1653-4b94-a4cc-ea5e10e70668" (UID: "8a0e97b3-1653-4b94-a4cc-ea5e10e70668"). InnerVolumeSpecName "kube-api-access-nrfwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:47:21.829578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.829546 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:21.829578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:21.829575 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrfwr\" (UniqueName: \"kubernetes.io/projected/8a0e97b3-1653-4b94-a4cc-ea5e10e70668-kube-api-access-nrfwr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:22.489118 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:22.489088 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" Nov 02 19:47:22.489266 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:22.489121 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t" event={"ID":"8a0e97b3-1653-4b94-a4cc-ea5e10e70668","Type":"ContainerDied","Data":"2d19ab65fb0ad95e99f8c427f5e51d0338813f94d057c8a9a95ecef162bf1acd"} Nov 02 19:47:22.489266 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:22.489176 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d19ab65fb0ad95e99f8c427f5e51d0338813f94d057c8a9a95ecef162bf1acd" Nov 02 19:47:25.044799 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:25.044766 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9"] Nov 02 19:47:25.092058 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:25.092028 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1116f601-gz6h9"] Nov 02 19:47:26.184349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:26.184277 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8546944b-e89c-4534-a504-3cf4ffc1fb60" path="/var/lib/kubelet/pods/8546944b-e89c-4534-a504-3cf4ffc1fb60/volumes" Nov 02 19:47:30.601361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.601324 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx"] Nov 02 19:47:30.601911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.601897 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8a0e97b3-1653-4b94-a4cc-ea5e10e70668" containerName="revoke-keys" Nov 02 19:47:30.601957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.601912 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0e97b3-1653-4b94-a4cc-ea5e10e70668" containerName="revoke-keys" Nov 02 19:47:30.602011 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.602003 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0e97b3-1653-4b94-a4cc-ea5e10e70668" containerName="revoke-keys" Nov 02 19:47:30.606849 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.606829 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.701000 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.700971 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-28623b\"/\"distinct-2-w2-28623b-kserve-self-signed-certs\"" Nov 02 19:47:30.705710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.705686 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx"] Nov 02 19:47:30.736791 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.736762 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tmp-dir\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.736952 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.736813 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvh5j\" (UniqueName: \"kubernetes.io/projected/c8801ef5-65bb-4798-a12c-6f769c84ba1a-kube-api-access-dvh5j\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.736952 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.736906 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-dshm\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.736952 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.736931 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tls-certs\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.737062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.737015 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-model-cache\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.737062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.737040 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-home\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.838312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.838252 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tmp-dir\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.838514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.838350 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvh5j\" (UniqueName: \"kubernetes.io/projected/c8801ef5-65bb-4798-a12c-6f769c84ba1a-kube-api-access-dvh5j\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.838807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.838404 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-dshm\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.838904 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.838861 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tls-certs\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.838904 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.838749 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tmp-dir\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.839014 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.838985 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-model-cache\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.839062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.839032 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-home\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.839455 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.839421 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-model-cache\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.839611 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.839588 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-home\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.841195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.841163 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-dshm\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.841907 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.841883 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tls-certs\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.891647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.891578 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvh5j\" (UniqueName: \"kubernetes.io/projected/c8801ef5-65bb-4798-a12c-6f769c84ba1a-kube-api-access-dvh5j\") pod \"distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:30.919126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:30.919089 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:31.354035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:31.353988 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx"] Nov 02 19:47:31.354155 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:31.354059 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8801ef5_65bb_4798_a12c_6f769c84ba1a.slice/crio-3f67e5969e26361372e593ad298642d23a5384e009e753365b8ac8f3e1f12e0a WatchSource:0}: Error finding container 3f67e5969e26361372e593ad298642d23a5384e009e753365b8ac8f3e1f12e0a: Status 404 returned error can't find the container with id 3f67e5969e26361372e593ad298642d23a5384e009e753365b8ac8f3e1f12e0a Nov 02 19:47:31.561661 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:31.561623 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" event={"ID":"c8801ef5-65bb-4798-a12c-6f769c84ba1a","Type":"ContainerStarted","Data":"e808a50e2856c3b9a55a92ae4fef4e945ce0d26a4430a360e54b270c2bf39442"} Nov 02 19:47:31.561877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:31.561676 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" event={"ID":"c8801ef5-65bb-4798-a12c-6f769c84ba1a","Type":"ContainerStarted","Data":"3f67e5969e26361372e593ad298642d23a5384e009e753365b8ac8f3e1f12e0a"} Nov 02 19:47:31.672872 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:31.672762 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podStartSLOduration=1.672747652 podStartE2EDuration="1.672747652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:31.669225323 +0000 UTC m=+1157.617881164" watchObservedRunningTime="2026-11-02 19:47:31.672747652 +0000 UTC m=+1157.621403490" Nov 02 19:47:31.920101 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:31.920066 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:34.597152 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.597118 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf"] Nov 02 19:47:34.601587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.601564 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791031 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.790984 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791250 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791112 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791250 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791180 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791354 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b8cb02-566c-4175-a961-c2d8a1f25316-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791387 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791496 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791541 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791577 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8b74\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-kube-api-access-w8b74\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.791635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.791614 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.801874 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.801839 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf"] Nov 02 19:47:34.824629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.824599 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh"] Nov 02 19:47:34.829018 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.828993 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.884662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.884582 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh"] Nov 02 19:47:34.892792 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.892762 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b8cb02-566c-4175-a961-c2d8a1f25316-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.892961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.892797 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.892961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.892890 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.892961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.892930 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.892964 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8b74\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-kube-api-access-w8b74\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.892994 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.893065 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.893095 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.893132 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.893662 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b8cb02-566c-4175-a961-c2d8a1f25316-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893777 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.893745 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.893850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.893692 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.894033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.894008 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.894462 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.894440 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.896992 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.896966 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.897158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.897142 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.946988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.946946 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.947598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.947572 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8b74\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-kube-api-access-w8b74\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:34.994572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.994536 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.994767 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.994579 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.994767 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.994676 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.994767 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.994708 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.994928 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.994808 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de86340-39ff-40bf-91f4-4dd3e8136c12-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.994928 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.994855 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9z89\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-kube-api-access-w9z89\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.995009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.994966 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.995055 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.995019 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:34.995093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:34.995069 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096283 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096367 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096400 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096419 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096452 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096478 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096528 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de86340-39ff-40bf-91f4-4dd3e8136c12-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096559 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9z89\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-kube-api-access-w9z89\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096677 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096973 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096874 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-credential-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.096973 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.096916 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-data\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.097082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.097055 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-socket\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.097133 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.097110 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-certs\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.097389 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.097368 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de86340-39ff-40bf-91f4-4dd3e8136c12-istiod-ca-cert\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.098809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.098787 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-envoy\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.099020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.099001 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-podinfo\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.149890 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.149822 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-token\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.149890 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.149850 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9z89\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-kube-api-access-w9z89\") pod \"e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.216693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.216662 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:35.445745 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.445660 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:35.643523 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:35.643478 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b8cb02_566c_4175_a961_c2d8a1f25316.slice/crio-e58e6edbf0ca994599cbf8d4211adfdba021a88e9e3290ca346f0aafa0af22c9 WatchSource:0}: Error finding container e58e6edbf0ca994599cbf8d4211adfdba021a88e9e3290ca346f0aafa0af22c9: Status 404 returned error can't find the container with id e58e6edbf0ca994599cbf8d4211adfdba021a88e9e3290ca346f0aafa0af22c9 Nov 02 19:47:35.643959 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.643745 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf"] Nov 02 19:47:35.646362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.646326 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:35.646484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.646393 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:35.646484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.646421 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:35.824917 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:35.824887 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de86340_39ff_40bf_91f4_4dd3e8136c12.slice/crio-6fb67f34d1f46ab36c969e461e3fde948ae1c503255288b0b317228cd35c01db WatchSource:0}: Error finding container 6fb67f34d1f46ab36c969e461e3fde948ae1c503255288b0b317228cd35c01db: Status 404 returned error can't find the container with id 6fb67f34d1f46ab36c969e461e3fde948ae1c503255288b0b317228cd35c01db Nov 02 19:47:35.825666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.825642 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh"] Nov 02 19:47:35.827864 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.827833 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:35.827965 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.827895 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:35.827965 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:35.827926 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:36.609561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:36.609418 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" event={"ID":"3de86340-39ff-40bf-91f4-4dd3e8136c12","Type":"ContainerStarted","Data":"da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b"} Nov 02 19:47:36.609561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:36.609507 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" event={"ID":"3de86340-39ff-40bf-91f4-4dd3e8136c12","Type":"ContainerStarted","Data":"6fb67f34d1f46ab36c969e461e3fde948ae1c503255288b0b317228cd35c01db"} Nov 02 19:47:36.611519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:36.611489 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" event={"ID":"80b8cb02-566c-4175-a961-c2d8a1f25316","Type":"ContainerStarted","Data":"8c766fcbb1f7d58389b728c9c11e70e8b9c24741bf0d30b4628f45e186038d0f"} Nov 02 19:47:36.611683 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:36.611542 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" event={"ID":"80b8cb02-566c-4175-a961-c2d8a1f25316","Type":"ContainerStarted","Data":"e58e6edbf0ca994599cbf8d4211adfdba021a88e9e3290ca346f0aafa0af22c9"} Nov 02 19:47:36.783323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:36.783106 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podStartSLOduration=2.783085974 podStartE2EDuration="2.783085974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.777804665 +0000 UTC m=+1162.726460510" watchObservedRunningTime="2026-11-02 19:47:36.783085974 +0000 UTC m=+1162.731741814" Nov 02 19:47:37.218077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:37.217523 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:37.219365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:37.219330 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 19:47:37.219495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:37.219390 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 19:47:37.449592 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:37.449551 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:37.449981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:37.449959 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 19:47:37.450461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:37.450162 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 19:47:38.217603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:38.217566 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 19:47:38.218009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:38.217617 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 19:47:38.448165 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:38.447658 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 19:47:38.448165 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:38.447721 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 19:47:39.217003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:39.216927 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 19:47:39.217003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:39.216990 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 19:47:39.446779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:39.446738 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 19:47:39.447425 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:39.447387 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 19:47:40.049323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.046623 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podStartSLOduration=6.046590333 podStartE2EDuration="6.046590333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:36.908239103 +0000 UTC m=+1162.856894942" watchObservedRunningTime="2026-11-02 19:47:40.046590333 +0000 UTC m=+1165.995246173" Nov 02 19:47:40.054086 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.054054 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d"] Nov 02 19:47:40.108713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.108674 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94fc97cf-9pd9d"] Nov 02 19:47:40.213250 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.213217 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18bd168-3fee-47b4-b464-ee294f996274" path="/var/lib/kubelet/pods/c18bd168-3fee-47b4-b464-ee294f996274/volumes" Nov 02 19:47:40.217734 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.217698 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 19:47:40.217925 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.217757 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 19:47:40.452379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.449680 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 19:47:40.452379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.449744 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 19:47:40.919654 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.919609 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:40.934215 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:40.934183 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:41.217429 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:41.217342 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 19:47:41.217429 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:41.217397 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 19:47:41.454329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:41.448673 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 19:47:41.454329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:41.448738 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 19:47:41.722424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:41.722378 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:47:42.217406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.217325 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 19:47:42.217406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.217390 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 19:47:42.446054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.446012 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 19:47:42.446229 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.446087 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 19:47:42.553454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.553411 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7"] Nov 02 19:47:42.559854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.559819 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.610106 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.610074 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:47:42.610487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.610466 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:47:42.611241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.610694 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:47:42.616900 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.616865 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7"] Nov 02 19:47:42.648638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.648601 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8w7\" (UniqueName: \"kubernetes.io/projected/4c625a86-1365-4c4b-8c72-17c4ce8d71af-kube-api-access-nk8w7\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.648884 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.648861 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.649202 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.649182 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.651350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.651223 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:47:42.760320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.752385 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.760320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.752493 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8w7\" (UniqueName: \"kubernetes.io/projected/4c625a86-1365-4c4b-8c72-17c4ce8d71af-kube-api-access-nk8w7\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.760320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.752543 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.760320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.755258 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.783324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.762313 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.807765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.807679 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8w7\" (UniqueName: \"kubernetes.io/projected/4c625a86-1365-4c4b-8c72-17c4ce8d71af-kube-api-access-nk8w7\") pod \"maas-api-revoke-sub-52f1974b-ljvc7\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:42.881102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:42.880866 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:43.347914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.347875 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7"] Nov 02 19:47:43.370053 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:43.370016 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c625a86_1365_4c4b_8c72_17c4ce8d71af.slice/crio-0453be449db4a7c96900faeb9783a438d7ee532afdc77c374e4d5b6dbed7e370 WatchSource:0}: Error finding container 0453be449db4a7c96900faeb9783a438d7ee532afdc77c374e4d5b6dbed7e370: Status 404 returned error can't find the container with id 0453be449db4a7c96900faeb9783a438d7ee532afdc77c374e4d5b6dbed7e370 Nov 02 19:47:43.695113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.693512 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:43.695113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.694961 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:43.698097 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.698077 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 19:47:43.750488 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.750447 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" event={"ID":"4c625a86-1365-4c4b-8c72-17c4ce8d71af","Type":"ContainerStarted","Data":"0453be449db4a7c96900faeb9783a438d7ee532afdc77c374e4d5b6dbed7e370"} Nov 02 19:47:43.890280 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.890250 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:43.890906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.890840 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:43.891532 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:43.891510 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 19:47:44.832379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:44.827615 4966 generic.go:356] "Generic (PLEG): container finished" podID="4c625a86-1365-4c4b-8c72-17c4ce8d71af" containerID="53fc8dfde2e5e95dc43ac6c7b60090304854e6580527160808bc0a6cb466f225" exitCode=0 Nov 02 19:47:44.832379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:44.828867 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" event={"ID":"4c625a86-1365-4c4b-8c72-17c4ce8d71af","Type":"ContainerDied","Data":"53fc8dfde2e5e95dc43ac6c7b60090304854e6580527160808bc0a6cb466f225"} Nov 02 19:47:44.894321 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:47:44.893777 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:47:46.432621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.428573 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:46.519091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.519057 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-xks-ca\") pod \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " Nov 02 19:47:46.519258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.519185 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-openshift-ca\") pod \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " Nov 02 19:47:46.519258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.519215 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c625a86-1365-4c4b-8c72-17c4ce8d71af-kube-api-access-nk8w7\" (UniqueName: \"kubernetes.io/projected/4c625a86-1365-4c4b-8c72-17c4ce8d71af-kube-api-access-nk8w7\") pod \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\" (UID: \"4c625a86-1365-4c4b-8c72-17c4ce8d71af\") " Nov 02 19:47:46.519652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.519615 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-openshift-ca" pod "4c625a86-1365-4c4b-8c72-17c4ce8d71af" (UID: "4c625a86-1365-4c4b-8c72-17c4ce8d71af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:47:46.521423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.521394 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c625a86-1365-4c4b-8c72-17c4ce8d71af-kube-api-access-nk8w7" pod "4c625a86-1365-4c4b-8c72-17c4ce8d71af" (UID: "4c625a86-1365-4c4b-8c72-17c4ce8d71af"). InnerVolumeSpecName "kube-api-access-nk8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:47:46.521646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.521629 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-xks-ca" pod "4c625a86-1365-4c4b-8c72-17c4ce8d71af" (UID: "4c625a86-1365-4c4b-8c72-17c4ce8d71af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:47:46.620463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.620425 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:46.620463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.620456 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk8w7\" (UniqueName: \"kubernetes.io/projected/4c625a86-1365-4c4b-8c72-17c4ce8d71af-kube-api-access-nk8w7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:46.620463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.620468 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c625a86-1365-4c4b-8c72-17c4ce8d71af-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:46.873471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.873432 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" Nov 02 19:47:46.873641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.873476 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7" event={"ID":"4c625a86-1365-4c4b-8c72-17c4ce8d71af","Type":"ContainerDied","Data":"0453be449db4a7c96900faeb9783a438d7ee532afdc77c374e4d5b6dbed7e370"} Nov 02 19:47:46.873641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:46.873534 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0453be449db4a7c96900faeb9783a438d7ee532afdc77c374e4d5b6dbed7e370" Nov 02 19:47:49.697617 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.697581 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z"] Nov 02 19:47:49.698239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.698224 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4c625a86-1365-4c4b-8c72-17c4ce8d71af" containerName="revoke-keys" Nov 02 19:47:49.698239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.698240 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c625a86-1365-4c4b-8c72-17c4ce8d71af" containerName="revoke-keys" Nov 02 19:47:49.698350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.698340 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c625a86-1365-4c4b-8c72-17c4ce8d71af" containerName="revoke-keys" Nov 02 19:47:49.702279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.702261 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.762625 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.762584 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.762851 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.762738 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.762851 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.762796 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.762961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.762879 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.763056 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.763027 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhljp\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-kube-api-access-vhljp\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.763107 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.763076 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.763159 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.763141 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.763238 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.763224 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.763292 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.763268 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.808163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.808128 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z"] Nov 02 19:47:49.865005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.864965 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865025 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865080 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865141 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhljp\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-kube-api-access-vhljp\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865172 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865196 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865231 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865273 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865422 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865422 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-data\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865581 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.865652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.865613 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-credential-socket\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.866071 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.866044 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-certs\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.866444 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.866417 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istiod-ca-cert\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.867812 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.867790 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-envoy\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.868028 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.868006 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-podinfo\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.919214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.919180 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-token\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:49.919385 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:49.919353 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhljp\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-kube-api-access-vhljp\") pod \"e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:50.016838 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.016810 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:50.447434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.447341 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z"] Nov 02 19:47:50.481217 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:50.481185 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90683c29_991f_4ed9_87e9_05d7b79d7aa1.slice/crio-5cb7dd06e248b19a462cac10e09f3dd4dc798c0aed060c4ebd8ceeb5fc48b3c8 WatchSource:0}: Error finding container 5cb7dd06e248b19a462cac10e09f3dd4dc798c0aed060c4ebd8ceeb5fc48b3c8: Status 404 returned error can't find the container with id 5cb7dd06e248b19a462cac10e09f3dd4dc798c0aed060c4ebd8ceeb5fc48b3c8 Nov 02 19:47:50.484176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.484139 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:50.484274 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.484229 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:50.484370 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.484329 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:47:50.594467 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.594432 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn"] Nov 02 19:47:50.600045 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.600019 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.651477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.651442 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:47:50.651856 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.651831 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:47:50.652555 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.652195 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:47:50.656708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.656682 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn"] Nov 02 19:47:50.682189 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.682086 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.682374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.682286 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.682374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.682366 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tptr\" (UniqueName: \"kubernetes.io/projected/565b1745-66c6-42ff-9252-afcca555a0eb-kube-api-access-8tptr\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.693867 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.693840 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:47:50.783574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.783535 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.784060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.783627 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tptr\" (UniqueName: \"kubernetes.io/projected/565b1745-66c6-42ff-9252-afcca555a0eb-kube-api-access-8tptr\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.784060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.783727 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.784710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.784678 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.786106 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.786082 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.837010 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.836978 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tptr\" (UniqueName: \"kubernetes.io/projected/565b1745-66c6-42ff-9252-afcca555a0eb-kube-api-access-8tptr\") pod \"maas-api-revoke-sub-59f44951-gwfpn\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:50.912119 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.912084 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" event={"ID":"90683c29-991f-4ed9-87e9-05d7b79d7aa1","Type":"ContainerStarted","Data":"74866f1dab6bf030963a343ebe8dc302de96e215af95ea11c8e52f88edd39aa2"} Nov 02 19:47:50.912282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.912138 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" event={"ID":"90683c29-991f-4ed9-87e9-05d7b79d7aa1","Type":"ContainerStarted","Data":"5cb7dd06e248b19a462cac10e09f3dd4dc798c0aed060c4ebd8ceeb5fc48b3c8"} Nov 02 19:47:50.919885 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:50.919850 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:51.017724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.017663 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:51.019083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.019053 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 19:47:51.019215 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.019113 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 19:47:51.024581 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.024533 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podStartSLOduration=2.024521547 podStartE2EDuration="2.024521547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:51.021424752 +0000 UTC m=+1176.970080592" watchObservedRunningTime="2026-11-02 19:47:51.024521547 +0000 UTC m=+1176.973177386" Nov 02 19:47:51.057449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.057372 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj"] Nov 02 19:47:51.063720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.063692 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7534a3f7-c2lgj"] Nov 02 19:47:51.314018 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.313987 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn"] Nov 02 19:47:51.315477 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:47:51.315449 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565b1745_66c6_42ff_9252_afcca555a0eb.slice/crio-ee0ceceedf83bf5a1bc48d6b2a2017e5228337a83e11b56048bedce2ea80ddab WatchSource:0}: Error finding container ee0ceceedf83bf5a1bc48d6b2a2017e5228337a83e11b56048bedce2ea80ddab: Status 404 returned error can't find the container with id ee0ceceedf83bf5a1bc48d6b2a2017e5228337a83e11b56048bedce2ea80ddab Nov 02 19:47:51.926660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.926558 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" event={"ID":"565b1745-66c6-42ff-9252-afcca555a0eb","Type":"ContainerStarted","Data":"9038d58cbae70d40e08d70797f7c1b6b9f39e0ff69c25bd44fff625f62fca272"} Nov 02 19:47:51.926660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:51.926638 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" event={"ID":"565b1745-66c6-42ff-9252-afcca555a0eb","Type":"ContainerStarted","Data":"ee0ceceedf83bf5a1bc48d6b2a2017e5228337a83e11b56048bedce2ea80ddab"} Nov 02 19:47:52.017206 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.017169 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 19:47:52.017382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.017224 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 19:47:52.032737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.032690 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" podStartSLOduration=2.032677881 podStartE2EDuration="2.032677881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:52.032563437 +0000 UTC m=+1177.981219275" watchObservedRunningTime="2026-11-02 19:47:52.032677881 +0000 UTC m=+1177.981333720" Nov 02 19:47:52.189819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.189728 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89dab0b7-2add-4566-b462-ea938eb712f2" path="/var/lib/kubelet/pods/89dab0b7-2add-4566-b462-ea938eb712f2/volumes" Nov 02 19:47:52.307054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.307018 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k"] Nov 02 19:47:52.311761 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.311738 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.379442 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.379396 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k"] Nov 02 19:47:52.405575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.405541 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8g4\" (UniqueName: \"kubernetes.io/projected/448a9154-e7f4-41fc-a523-ad07e46f4a75-kube-api-access-mt8g4\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.405792 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.405771 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.405908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.405893 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.506978 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.506940 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.507146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.507025 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.507146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.507060 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8g4\" (UniqueName: \"kubernetes.io/projected/448a9154-e7f4-41fc-a523-ad07e46f4a75-kube-api-access-mt8g4\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.507756 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.507727 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.509303 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.509281 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.576675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.576634 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8g4\" (UniqueName: \"kubernetes.io/projected/448a9154-e7f4-41fc-a523-ad07e46f4a75-kube-api-access-mt8g4\") pod \"maas-api-revoke-sub-e6d15206-7vx9k\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.626401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.626363 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:52.986919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.986869 4966 generic.go:356] "Generic (PLEG): container finished" podID="565b1745-66c6-42ff-9252-afcca555a0eb" containerID="9038d58cbae70d40e08d70797f7c1b6b9f39e0ff69c25bd44fff625f62fca272" exitCode=0 Nov 02 19:47:52.987424 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:52.987073 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" event={"ID":"565b1745-66c6-42ff-9252-afcca555a0eb","Type":"ContainerDied","Data":"9038d58cbae70d40e08d70797f7c1b6b9f39e0ff69c25bd44fff625f62fca272"} Nov 02 19:47:53.017842 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:53.017811 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 19:47:53.017971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:53.017859 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 19:47:53.082167 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:53.081729 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k"] Nov 02 19:47:53.997166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:53.997110 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" event={"ID":"448a9154-e7f4-41fc-a523-ad07e46f4a75","Type":"ContainerStarted","Data":"a0ee2ae818fbb3e115db770f8d1c36e75b5ca41507e6a5c6b8edd8a49f16df4b"} Nov 02 19:47:53.997580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:53.997193 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" event={"ID":"448a9154-e7f4-41fc-a523-ad07e46f4a75","Type":"ContainerStarted","Data":"70738f19e70d89628c34816c791d7c447ae0c7e3429719b27948b53db879a224"} Nov 02 19:47:54.017550 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.017522 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 19:47:54.017705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.017569 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 19:47:54.115180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.115115 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" podStartSLOduration=2.115100139 podStartE2EDuration="2.115100139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:54.110422108 +0000 UTC m=+1180.059077946" watchObservedRunningTime="2026-11-02 19:47:54.115100139 +0000 UTC m=+1180.063755977" Nov 02 19:47:54.219396 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.219369 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:54.331818 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.331786 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-openshift-ca\") pod \"565b1745-66c6-42ff-9252-afcca555a0eb\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " Nov 02 19:47:54.331997 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.331978 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-xks-ca\") pod \"565b1745-66c6-42ff-9252-afcca555a0eb\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " Nov 02 19:47:54.332093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.332080 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565b1745-66c6-42ff-9252-afcca555a0eb-kube-api-access-8tptr\" (UniqueName: \"kubernetes.io/projected/565b1745-66c6-42ff-9252-afcca555a0eb-kube-api-access-8tptr\") pod \"565b1745-66c6-42ff-9252-afcca555a0eb\" (UID: \"565b1745-66c6-42ff-9252-afcca555a0eb\") " Nov 02 19:47:54.332187 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.332161 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-openshift-ca" pod "565b1745-66c6-42ff-9252-afcca555a0eb" (UID: "565b1745-66c6-42ff-9252-afcca555a0eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:47:54.332639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.332614 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:54.334351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.334216 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-xks-ca" pod "565b1745-66c6-42ff-9252-afcca555a0eb" (UID: "565b1745-66c6-42ff-9252-afcca555a0eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:47:54.334351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.334322 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565b1745-66c6-42ff-9252-afcca555a0eb-kube-api-access-8tptr" pod "565b1745-66c6-42ff-9252-afcca555a0eb" (UID: "565b1745-66c6-42ff-9252-afcca555a0eb"). InnerVolumeSpecName "kube-api-access-8tptr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:47:54.434047 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.434005 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565b1745-66c6-42ff-9252-afcca555a0eb-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:54.434047 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:54.434042 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tptr\" (UniqueName: \"kubernetes.io/projected/565b1745-66c6-42ff-9252-afcca555a0eb-kube-api-access-8tptr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:55.008273 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:55.008243 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" Nov 02 19:47:55.008723 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:55.008284 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn" event={"ID":"565b1745-66c6-42ff-9252-afcca555a0eb","Type":"ContainerDied","Data":"ee0ceceedf83bf5a1bc48d6b2a2017e5228337a83e11b56048bedce2ea80ddab"} Nov 02 19:47:55.008723 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:55.008364 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee0ceceedf83bf5a1bc48d6b2a2017e5228337a83e11b56048bedce2ea80ddab" Nov 02 19:47:55.010363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:55.010338 4966 generic.go:356] "Generic (PLEG): container finished" podID="448a9154-e7f4-41fc-a523-ad07e46f4a75" containerID="a0ee2ae818fbb3e115db770f8d1c36e75b5ca41507e6a5c6b8edd8a49f16df4b" exitCode=0 Nov 02 19:47:55.010500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:55.010382 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" event={"ID":"448a9154-e7f4-41fc-a523-ad07e46f4a75","Type":"ContainerDied","Data":"a0ee2ae818fbb3e115db770f8d1c36e75b5ca41507e6a5c6b8edd8a49f16df4b"} Nov 02 19:47:55.017179 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:55.017150 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 19:47:55.017240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:55.017195 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 19:47:56.027714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.027677 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 19:47:56.028465 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.028199 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 19:47:56.272970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.272942 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:56.384441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.384361 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448a9154-e7f4-41fc-a523-ad07e46f4a75-kube-api-access-mt8g4\" (UniqueName: \"kubernetes.io/projected/448a9154-e7f4-41fc-a523-ad07e46f4a75-kube-api-access-mt8g4\") pod \"448a9154-e7f4-41fc-a523-ad07e46f4a75\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " Nov 02 19:47:56.384613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.384555 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-xks-ca\") pod \"448a9154-e7f4-41fc-a523-ad07e46f4a75\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " Nov 02 19:47:56.384613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.384586 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-openshift-ca\") pod \"448a9154-e7f4-41fc-a523-ad07e46f4a75\" (UID: \"448a9154-e7f4-41fc-a523-ad07e46f4a75\") " Nov 02 19:47:56.385534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.385482 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-openshift-ca" pod "448a9154-e7f4-41fc-a523-ad07e46f4a75" (UID: "448a9154-e7f4-41fc-a523-ad07e46f4a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:47:56.390169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.387936 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-xks-ca" pod "448a9154-e7f4-41fc-a523-ad07e46f4a75" (UID: "448a9154-e7f4-41fc-a523-ad07e46f4a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:47:56.397712 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.397684 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448a9154-e7f4-41fc-a523-ad07e46f4a75-kube-api-access-mt8g4" pod "448a9154-e7f4-41fc-a523-ad07e46f4a75" (UID: "448a9154-e7f4-41fc-a523-ad07e46f4a75"). InnerVolumeSpecName "kube-api-access-mt8g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:47:56.486566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.486530 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:56.486566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.486561 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448a9154-e7f4-41fc-a523-ad07e46f4a75-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:56.486566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:56.486571 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt8g4\" (UniqueName: \"kubernetes.io/projected/448a9154-e7f4-41fc-a523-ad07e46f4a75-kube-api-access-mt8g4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:47:57.018165 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:57.018127 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 19:47:57.018354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:57.018194 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 19:47:57.042141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:57.042103 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" Nov 02 19:47:57.042642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:57.042143 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k" event={"ID":"448a9154-e7f4-41fc-a523-ad07e46f4a75","Type":"ContainerDied","Data":"70738f19e70d89628c34816c791d7c447ae0c7e3429719b27948b53db879a224"} Nov 02 19:47:57.042642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:57.042191 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70738f19e70d89628c34816c791d7c447ae0c7e3429719b27948b53db879a224" Nov 02 19:47:58.077573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:58.077543 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:58.078049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:58.078025 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:47:58.078876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:47:58.078857 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 19:48:04.941675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.941636 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd"] Nov 02 19:48:04.942636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.942607 4966 cpu_manager.go:414] "containerMap: removing container" podUID="448a9154-e7f4-41fc-a523-ad07e46f4a75" containerName="revoke-keys" Nov 02 19:48:04.942838 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.942817 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448a9154-e7f4-41fc-a523-ad07e46f4a75" containerName="revoke-keys" Nov 02 19:48:04.942984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.942970 4966 cpu_manager.go:414] "containerMap: removing container" podUID="565b1745-66c6-42ff-9252-afcca555a0eb" containerName="revoke-keys" Nov 02 19:48:04.943079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.943058 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565b1745-66c6-42ff-9252-afcca555a0eb" containerName="revoke-keys" Nov 02 19:48:04.943291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.943274 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="448a9154-e7f4-41fc-a523-ad07e46f4a75" containerName="revoke-keys" Nov 02 19:48:04.943409 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.943317 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="565b1745-66c6-42ff-9252-afcca555a0eb" containerName="revoke-keys" Nov 02 19:48:04.947249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.947222 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.988249 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.988214 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7"] Nov 02 19:48:04.993122 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.993093 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:04.995077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995048 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550e9d0-4fb2-451e-a67d-5996dae13934-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.995221 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995085 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.995221 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995175 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.995221 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995203 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8ss\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-kube-api-access-zg8ss\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.995509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995260 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.995618 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995465 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.995681 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995663 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.995882 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995857 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:04.996011 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:04.995989 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.034711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.034666 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd"] Nov 02 19:48:05.054352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.054277 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd"] Nov 02 19:48:05.063323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.063278 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f021618-khrnd"] Nov 02 19:48:05.065908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.065880 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7"] Nov 02 19:48:05.097513 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097474 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.097667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097541 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.097667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097608 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.097667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097648 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.097860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097839 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.097905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097878 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.097953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097923 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550e9d0-4fb2-451e-a67d-5996dae13934-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.098011 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.097961 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.098059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098046 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfb6\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-kube-api-access-rsfb6\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.098108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098055 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.098108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098099 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098465 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098504 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8ss\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-kube-api-access-zg8ss\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098555 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098671 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098678 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098801 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098882 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.098961 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.099631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.099016 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.100166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.099650 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550e9d0-4fb2-451e-a67d-5996dae13934-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.100258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.100238 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.100357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.100292 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.101190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.101168 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.102254 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.102230 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.153000 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.152969 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.153135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.153110 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8ss\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-kube-api-access-zg8ss\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.199841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.199774 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.199975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.199856 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.199975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.199885 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.199975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.199956 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200092 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.199993 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200167 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200139 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200259 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200371 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-credential-socket\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200371 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200461 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfb6\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-kube-api-access-rsfb6\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200461 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-certs\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200537 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.200858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.200835 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-data\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.201070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.201050 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istiod-ca-cert\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.202562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.202539 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-envoy\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.202950 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.202932 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-podinfo\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.256352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.256320 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfb6\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-kube-api-access-rsfb6\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.256352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.256340 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-token\") pod \"e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.270895 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.270864 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:05.313527 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.313493 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:05.709627 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:05.709543 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4550e9d0_4fb2_451e_a67d_5996dae13934.slice/crio-7a9940cd64e381dcfad26f70b509667d0f3ffd64a230289fc8a0b533dafb29ed WatchSource:0}: Error finding container 7a9940cd64e381dcfad26f70b509667d0f3ffd64a230289fc8a0b533dafb29ed: Status 404 returned error can't find the container with id 7a9940cd64e381dcfad26f70b509667d0f3ffd64a230289fc8a0b533dafb29ed Nov 02 19:48:05.711336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.710840 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd"] Nov 02 19:48:05.715019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.714982 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:05.715149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.715068 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:05.715149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.715107 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:05.752840 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:05.752800 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda253b41d_9ef0_4447_a246_fd2579ef1d5f.slice/crio-9045a144c9f14ef74c598117cb68bb35f16c092bda5a45c8671e0d15f0921f4d WatchSource:0}: Error finding container 9045a144c9f14ef74c598117cb68bb35f16c092bda5a45c8671e0d15f0921f4d: Status 404 returned error can't find the container with id 9045a144c9f14ef74c598117cb68bb35f16c092bda5a45c8671e0d15f0921f4d Nov 02 19:48:05.754012 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.753985 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7"] Nov 02 19:48:05.755622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.755590 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:05.755691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.755658 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:05.755691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:05.755686 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:06.127618 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.127576 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" event={"ID":"4550e9d0-4fb2-451e-a67d-5996dae13934","Type":"ContainerStarted","Data":"faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578"} Nov 02 19:48:06.128196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.128172 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" event={"ID":"4550e9d0-4fb2-451e-a67d-5996dae13934","Type":"ContainerStarted","Data":"7a9940cd64e381dcfad26f70b509667d0f3ffd64a230289fc8a0b533dafb29ed"} Nov 02 19:48:06.130555 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.130526 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" event={"ID":"a253b41d-9ef0-4447-a246-fd2579ef1d5f","Type":"ContainerStarted","Data":"89b1c6b01f6f80d33744534f8489f35d23658d48a24dfe66b087080c34c6dc4f"} Nov 02 19:48:06.130690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.130582 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" event={"ID":"a253b41d-9ef0-4447-a246-fd2579ef1d5f","Type":"ContainerStarted","Data":"9045a144c9f14ef74c598117cb68bb35f16c092bda5a45c8671e0d15f0921f4d"} Nov 02 19:48:06.190192 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.190154 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4167e9-46f7-4dd4-b94e-111429636de1" path="/var/lib/kubelet/pods/4b4167e9-46f7-4dd4-b94e-111429636de1/volumes" Nov 02 19:48:06.240438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.240375 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podStartSLOduration=2.240358339 podStartE2EDuration="2.240358339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:06.238772272 +0000 UTC m=+1192.187428111" watchObservedRunningTime="2026-11-02 19:48:06.240358339 +0000 UTC m=+1192.189014178" Nov 02 19:48:06.271497 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.271463 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:06.272970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.272939 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 19:48:06.273099 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.273002 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 19:48:06.314549 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.314412 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:06.315750 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.315727 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 19:48:06.315862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.315784 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 19:48:06.353696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:06.353629 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podStartSLOduration=2.35361502 podStartE2EDuration="2.35361502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:06.348405602 +0000 UTC m=+1192.297061438" watchObservedRunningTime="2026-11-02 19:48:06.35361502 +0000 UTC m=+1192.302270859" Nov 02 19:48:07.271732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:07.271695 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 19:48:07.272356 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:07.272319 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 19:48:07.314021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:07.313986 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 19:48:07.314275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:07.314249 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 19:48:08.271710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.271671 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 19:48:08.272002 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.271976 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 19:48:08.314017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.313889 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 19:48:08.314017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.313944 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 19:48:08.900397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.900346 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls"] Nov 02 19:48:08.908933 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.908902 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:08.956896 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.956855 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls"] Nov 02 19:48:08.982331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.980579 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:48:08.982331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.980739 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:48:08.982331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.980774 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:48:08.982331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:08.981473 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:48:09.015326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.014055 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgw5q\" (UniqueName: \"kubernetes.io/projected/37f4fd51-e61d-445b-92ce-2304977e1892-kube-api-access-jgw5q\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.015326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.014247 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.015326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.014375 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.115268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.115234 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.115435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.115342 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.115435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.115432 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgw5q\" (UniqueName: \"kubernetes.io/projected/37f4fd51-e61d-445b-92ce-2304977e1892-kube-api-access-jgw5q\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.116178 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.116150 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.117842 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.117815 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.199367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.198841 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgw5q\" (UniqueName: \"kubernetes.io/projected/37f4fd51-e61d-445b-92ce-2304977e1892-kube-api-access-jgw5q\") pod \"maas-api-revoke-sub-aa2132e8-p8kls\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.227367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.227331 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:09.305322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.299028 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 19:48:09.305322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.299099 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 19:48:09.330826 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.330790 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 19:48:09.330972 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.330847 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 19:48:09.830191 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:09.830139 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f4fd51_e61d_445b_92ce_2304977e1892.slice/crio-cea799d5901b23b5a930641c7fe0aa9851a8dd6ac89ce571947a04842d83dac4 WatchSource:0}: Error finding container cea799d5901b23b5a930641c7fe0aa9851a8dd6ac89ce571947a04842d83dac4: Status 404 returned error can't find the container with id cea799d5901b23b5a930641c7fe0aa9851a8dd6ac89ce571947a04842d83dac4 Nov 02 19:48:09.831425 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:09.831331 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls"] Nov 02 19:48:10.082020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.081885 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn"] Nov 02 19:48:10.101359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.101243 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6403179-69txn"] Nov 02 19:48:10.210346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.210218 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67582cc-de5b-42a8-8bc0-46e5f59614b5" path="/var/lib/kubelet/pods/b67582cc-de5b-42a8-8bc0-46e5f59614b5/volumes" Nov 02 19:48:10.299345 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.299309 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 19:48:10.299662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.299614 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 19:48:10.322700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.322169 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 19:48:10.322700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.322225 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 19:48:10.336417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:10.336288 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" event={"ID":"37f4fd51-e61d-445b-92ce-2304977e1892","Type":"ContainerStarted","Data":"cea799d5901b23b5a930641c7fe0aa9851a8dd6ac89ce571947a04842d83dac4"} Nov 02 19:48:11.316252 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:11.316131 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 19:48:11.316252 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:11.316210 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 19:48:11.316552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:11.316237 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 19:48:11.316552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:11.316533 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 19:48:11.415120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:11.414492 4966 generic.go:356] "Generic (PLEG): container finished" podID="37f4fd51-e61d-445b-92ce-2304977e1892" containerID="753703e9fee0f505ad5d757cad10103f5a2ca698cf53e63edeaafd32047a4c5c" exitCode=0 Nov 02 19:48:11.415120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:11.414759 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" event={"ID":"37f4fd51-e61d-445b-92ce-2304977e1892","Type":"ContainerDied","Data":"753703e9fee0f505ad5d757cad10103f5a2ca698cf53e63edeaafd32047a4c5c"} Nov 02 19:48:12.272197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.272151 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 19:48:12.272401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.272225 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 19:48:12.314040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.314003 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 02 19:48:12.314188 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.314069 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 02 19:48:12.623084 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.623029 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:12.654970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.654941 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-openshift-ca\") pod \"37f4fd51-e61d-445b-92ce-2304977e1892\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " Nov 02 19:48:12.655181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.655161 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-xks-ca\") pod \"37f4fd51-e61d-445b-92ce-2304977e1892\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " Nov 02 19:48:12.655359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.655338 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f4fd51-e61d-445b-92ce-2304977e1892-kube-api-access-jgw5q\" (UniqueName: \"kubernetes.io/projected/37f4fd51-e61d-445b-92ce-2304977e1892-kube-api-access-jgw5q\") pod \"37f4fd51-e61d-445b-92ce-2304977e1892\" (UID: \"37f4fd51-e61d-445b-92ce-2304977e1892\") " Nov 02 19:48:12.656257 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.656231 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-openshift-ca" pod "37f4fd51-e61d-445b-92ce-2304977e1892" (UID: "37f4fd51-e61d-445b-92ce-2304977e1892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:12.658412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.658387 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f4fd51-e61d-445b-92ce-2304977e1892-kube-api-access-jgw5q" pod "37f4fd51-e61d-445b-92ce-2304977e1892" (UID: "37f4fd51-e61d-445b-92ce-2304977e1892"). InnerVolumeSpecName "kube-api-access-jgw5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:12.659490 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.659466 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-xks-ca" pod "37f4fd51-e61d-445b-92ce-2304977e1892" (UID: "37f4fd51-e61d-445b-92ce-2304977e1892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:12.767202 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.767161 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgw5q\" (UniqueName: \"kubernetes.io/projected/37f4fd51-e61d-445b-92ce-2304977e1892-kube-api-access-jgw5q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:12.767202 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.767203 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:12.767494 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:12.767220 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f4fd51-e61d-445b-92ce-2304977e1892-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.491321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.491266 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" event={"ID":"37f4fd51-e61d-445b-92ce-2304977e1892","Type":"ContainerDied","Data":"cea799d5901b23b5a930641c7fe0aa9851a8dd6ac89ce571947a04842d83dac4"} Nov 02 19:48:13.491583 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.491565 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cea799d5901b23b5a930641c7fe0aa9851a8dd6ac89ce571947a04842d83dac4" Nov 02 19:48:13.491734 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.491723 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls" Nov 02 19:48:13.605738 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.605701 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:13.606709 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.606651 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:13.607777 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.607726 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:48:13.635558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.635531 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:13.636463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.636439 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:13.637853 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.637821 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 19:48:13.897946 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.897859 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44"] Nov 02 19:48:13.898565 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.898542 4966 cpu_manager.go:414] "containerMap: removing container" podUID="37f4fd51-e61d-445b-92ce-2304977e1892" containerName="revoke-keys" Nov 02 19:48:13.898678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.898564 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f4fd51-e61d-445b-92ce-2304977e1892" containerName="revoke-keys" Nov 02 19:48:13.898792 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.898777 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f4fd51-e61d-445b-92ce-2304977e1892" containerName="revoke-keys" Nov 02 19:48:13.902698 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.902677 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:13.954140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.954108 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:48:13.954730 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.954707 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:48:13.955265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.954983 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:48:13.955395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.955065 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:48:13.983376 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:13.983340 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44"] Nov 02 19:48:14.000432 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.000384 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2mr\" (UniqueName: \"kubernetes.io/projected/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-kube-api-access-hq2mr\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.000599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.000445 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.000599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.000494 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.101704 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.101670 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2mr\" (UniqueName: \"kubernetes.io/projected/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-kube-api-access-hq2mr\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.101906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.101712 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.101906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.101750 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.102533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.102512 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.104098 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.104080 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.199971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.199886 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:48:14.205365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.205336 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2mr\" (UniqueName: \"kubernetes.io/projected/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-kube-api-access-hq2mr\") pod \"maas-api-revoke-sub-3694f937-tgr44\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.265217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.265183 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:48:14.267592 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.267563 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:14.405282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.405249 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:48:14.408880 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.408855 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:48:14.418207 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.418183 4966 scope.go:122] "RemoveContainer" containerID="8e62467113064e12921d934f5a29b6ce25f0ccffb158b89dd350ad579eb5825b" Nov 02 19:48:14.426114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.426090 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:48:14.430893 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.430868 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:48:14.641567 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:14.641533 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea38265_8a8b_4ee0_b0eb_21ea6c6d8676.slice/crio-842469024bb0c3552bf7ae9c61c25cd39f5bdc8f11ac3d764afefd8dce954c3a WatchSource:0}: Error finding container 842469024bb0c3552bf7ae9c61c25cd39f5bdc8f11ac3d764afefd8dce954c3a: Status 404 returned error can't find the container with id 842469024bb0c3552bf7ae9c61c25cd39f5bdc8f11ac3d764afefd8dce954c3a Nov 02 19:48:14.643892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.643864 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44"] Nov 02 19:48:14.681162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.681131 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64"] Nov 02 19:48:14.687013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.686991 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.709743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.709716 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.709973 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.709950 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.710029 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.709982 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qvk\" (UniqueName: \"kubernetes.io/projected/6a418928-2d73-43ca-a1cd-1eed624b7391-kube-api-access-c4qvk\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.744569 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.744531 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64"] Nov 02 19:48:14.778795 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:48:14.778758 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:48:14.811484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.811449 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.811668 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.811491 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qvk\" (UniqueName: \"kubernetes.io/projected/6a418928-2d73-43ca-a1cd-1eed624b7391-kube-api-access-c4qvk\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.811668 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.811586 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.812542 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.812513 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.815350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.815325 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:14.865217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:14.865184 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qvk\" (UniqueName: \"kubernetes.io/projected/6a418928-2d73-43ca-a1cd-1eed624b7391-kube-api-access-c4qvk\") pod \"maas-api-revoke-sub-1545a833-x9v64\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:15.000713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:15.000680 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:15.394743 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:15.394701 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a418928_2d73_43ca_a1cd_1eed624b7391.slice/crio-30590505fc136590c95d6d2ce3163721c4e2035ef8fb2f4f551c05e2c0172a47 WatchSource:0}: Error finding container 30590505fc136590c95d6d2ce3163721c4e2035ef8fb2f4f551c05e2c0172a47: Status 404 returned error can't find the container with id 30590505fc136590c95d6d2ce3163721c4e2035ef8fb2f4f551c05e2c0172a47 Nov 02 19:48:15.395148 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:15.395127 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64"] Nov 02 19:48:15.513422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:15.513382 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" event={"ID":"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676","Type":"ContainerStarted","Data":"2baa328b432229497eda8f81629c51ff9ba8550b4ef5b6c787895ff163ca64ce"} Nov 02 19:48:15.513630 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:15.513449 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" event={"ID":"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676","Type":"ContainerStarted","Data":"842469024bb0c3552bf7ae9c61c25cd39f5bdc8f11ac3d764afefd8dce954c3a"} Nov 02 19:48:15.516843 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:15.516807 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" event={"ID":"6a418928-2d73-43ca-a1cd-1eed624b7391","Type":"ContainerStarted","Data":"30590505fc136590c95d6d2ce3163721c4e2035ef8fb2f4f551c05e2c0172a47"} Nov 02 19:48:15.637550 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:15.637488 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" podStartSLOduration=2.637474394 podStartE2EDuration="2.637474394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:15.634993451 +0000 UTC m=+1201.583649291" watchObservedRunningTime="2026-11-02 19:48:15.637474394 +0000 UTC m=+1201.586130246" Nov 02 19:48:16.528540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:16.528510 4966 generic.go:356] "Generic (PLEG): container finished" podID="6a418928-2d73-43ca-a1cd-1eed624b7391" containerID="d4b64393ba0cdefa97fc729b4f8c65982ea0b875bfb7a21d8a85763c8421e938" exitCode=0 Nov 02 19:48:16.528948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:16.528592 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" event={"ID":"6a418928-2d73-43ca-a1cd-1eed624b7391","Type":"ContainerDied","Data":"d4b64393ba0cdefa97fc729b4f8c65982ea0b875bfb7a21d8a85763c8421e938"} Nov 02 19:48:16.530601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:16.530575 4966 generic.go:356] "Generic (PLEG): container finished" podID="1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" containerID="2baa328b432229497eda8f81629c51ff9ba8550b4ef5b6c787895ff163ca64ce" exitCode=0 Nov 02 19:48:16.530737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:16.530606 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" event={"ID":"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676","Type":"ContainerDied","Data":"2baa328b432229497eda8f81629c51ff9ba8550b4ef5b6c787895ff163ca64ce"} Nov 02 19:48:17.052580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.052537 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6"] Nov 02 19:48:17.061109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.061073 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ca2b5c-w6jf6"] Nov 02 19:48:17.779490 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.779462 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:17.784520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.784489 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:17.871915 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.871831 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-openshift-ca\") pod \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " Nov 02 19:48:17.871915 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.871887 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-xks-ca\") pod \"6a418928-2d73-43ca-a1cd-1eed624b7391\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " Nov 02 19:48:17.871915 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.871916 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-xks-ca\") pod \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " Nov 02 19:48:17.872186 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.871934 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a418928-2d73-43ca-a1cd-1eed624b7391-kube-api-access-c4qvk\" (UniqueName: \"kubernetes.io/projected/6a418928-2d73-43ca-a1cd-1eed624b7391-kube-api-access-c4qvk\") pod \"6a418928-2d73-43ca-a1cd-1eed624b7391\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " Nov 02 19:48:17.872186 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.871989 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-openshift-ca\") pod \"6a418928-2d73-43ca-a1cd-1eed624b7391\" (UID: \"6a418928-2d73-43ca-a1cd-1eed624b7391\") " Nov 02 19:48:17.872186 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.872152 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-kube-api-access-hq2mr\" (UniqueName: \"kubernetes.io/projected/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-kube-api-access-hq2mr\") pod \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\" (UID: \"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676\") " Nov 02 19:48:17.872392 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.872252 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-openshift-ca" pod "1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" (UID: "1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:17.872655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.872629 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-openshift-ca" pod "6a418928-2d73-43ca-a1cd-1eed624b7391" (UID: "6a418928-2d73-43ca-a1cd-1eed624b7391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:17.872861 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.872834 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:17.872861 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.872858 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:17.874284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.874257 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-xks-ca" pod "6a418928-2d73-43ca-a1cd-1eed624b7391" (UID: "6a418928-2d73-43ca-a1cd-1eed624b7391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:17.874580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.874550 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-xks-ca" pod "1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" (UID: "1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:17.874867 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.874843 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a418928-2d73-43ca-a1cd-1eed624b7391-kube-api-access-c4qvk" pod "6a418928-2d73-43ca-a1cd-1eed624b7391" (UID: "6a418928-2d73-43ca-a1cd-1eed624b7391"). InnerVolumeSpecName "kube-api-access-c4qvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:17.875059 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.875038 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-kube-api-access-hq2mr" pod "1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" (UID: "1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676"). InnerVolumeSpecName "kube-api-access-hq2mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:17.974479 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.974441 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq2mr\" (UniqueName: \"kubernetes.io/projected/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-kube-api-access-hq2mr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:17.974479 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.974474 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a418928-2d73-43ca-a1cd-1eed624b7391-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:17.974680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.974489 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:17.974680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:17.974501 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qvk\" (UniqueName: \"kubernetes.io/projected/6a418928-2d73-43ca-a1cd-1eed624b7391-kube-api-access-c4qvk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:18.185649 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:18.185614 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d801efa-5982-4805-a41d-cf64ed79f1e3" path="/var/lib/kubelet/pods/6d801efa-5982-4805-a41d-cf64ed79f1e3/volumes" Nov 02 19:48:18.552330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:18.552273 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" Nov 02 19:48:18.552330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:18.552288 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64" event={"ID":"6a418928-2d73-43ca-a1cd-1eed624b7391","Type":"ContainerDied","Data":"30590505fc136590c95d6d2ce3163721c4e2035ef8fb2f4f551c05e2c0172a47"} Nov 02 19:48:18.552553 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:18.552371 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30590505fc136590c95d6d2ce3163721c4e2035ef8fb2f4f551c05e2c0172a47" Nov 02 19:48:18.555057 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:18.554946 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" event={"ID":"1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676","Type":"ContainerDied","Data":"842469024bb0c3552bf7ae9c61c25cd39f5bdc8f11ac3d764afefd8dce954c3a"} Nov 02 19:48:18.555057 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:18.554991 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="842469024bb0c3552bf7ae9c61c25cd39f5bdc8f11ac3d764afefd8dce954c3a" Nov 02 19:48:18.555057 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:18.555048 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44" Nov 02 19:48:22.710380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.708251 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb"] Nov 02 19:48:22.710380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.709723 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6a418928-2d73-43ca-a1cd-1eed624b7391" containerName="revoke-keys" Nov 02 19:48:22.710380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.709747 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a418928-2d73-43ca-a1cd-1eed624b7391" containerName="revoke-keys" Nov 02 19:48:22.710380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.709822 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" containerName="revoke-keys" Nov 02 19:48:22.710380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.709831 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" containerName="revoke-keys" Nov 02 19:48:22.710380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.709987 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" containerName="revoke-keys" Nov 02 19:48:22.710380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.710005 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a418928-2d73-43ca-a1cd-1eed624b7391" containerName="revoke-keys" Nov 02 19:48:22.715692 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.715665 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.770551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.770520 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb"] Nov 02 19:48:22.807077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.807034 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.807267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.807201 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jglms\" (UniqueName: \"kubernetes.io/projected/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-kube-api-access-jglms\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.807689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.807659 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.811585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.811558 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.811585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.811565 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:48:22.811767 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.811653 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.815437 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.815418 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:48:22.908804 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.908773 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.908957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.908825 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.908957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.908918 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglms\" (UniqueName: \"kubernetes.io/projected/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-kube-api-access-jglms\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.909595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.909567 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.911283 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.911262 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:22.962040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:22.961957 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jglms\" (UniqueName: \"kubernetes.io/projected/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-kube-api-access-jglms\") pod \"maas-api-revoke-sub-f5c625c1-gp9hb\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:23.038939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:23.038907 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:23.415352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:23.415309 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb"] Nov 02 19:48:23.424140 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:23.424105 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e4ff5c_2443_45aa_afa4_83300ac39fc9.slice/crio-b0ff70417d82db756ef310baff0895e167c351a454ecd3af8b8e9aaaf0d172d8 WatchSource:0}: Error finding container b0ff70417d82db756ef310baff0895e167c351a454ecd3af8b8e9aaaf0d172d8: Status 404 returned error can't find the container with id b0ff70417d82db756ef310baff0895e167c351a454ecd3af8b8e9aaaf0d172d8 Nov 02 19:48:23.655120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:23.655088 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" event={"ID":"e5e4ff5c-2443-45aa-afa4-83300ac39fc9","Type":"ContainerStarted","Data":"b0ff70417d82db756ef310baff0895e167c351a454ecd3af8b8e9aaaf0d172d8"} Nov 02 19:48:24.046242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.046154 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q"] Nov 02 19:48:24.051397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.051370 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871dd1ce-c9b8q"] Nov 02 19:48:24.188231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.188191 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe3c9bb-e604-4042-aa48-752beaeda872" path="/var/lib/kubelet/pods/fbe3c9bb-e604-4042-aa48-752beaeda872/volumes" Nov 02 19:48:24.618562 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.618478 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5"] Nov 02 19:48:24.623173 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.623151 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.667001 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.666971 4966 generic.go:356] "Generic (PLEG): container finished" podID="e5e4ff5c-2443-45aa-afa4-83300ac39fc9" containerID="8ad5d4d8dffc106419cd5cc6b02a38771bc7b57a333d21ede248de4b66bd0564" exitCode=0 Nov 02 19:48:24.667142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.667063 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" event={"ID":"e5e4ff5c-2443-45aa-afa4-83300ac39fc9","Type":"ContainerDied","Data":"8ad5d4d8dffc106419cd5cc6b02a38771bc7b57a333d21ede248de4b66bd0564"} Nov 02 19:48:24.697663 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.697585 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5"] Nov 02 19:48:24.731755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.731711 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkt6\" (UniqueName: \"kubernetes.io/projected/947f465c-d8b4-4b83-af9f-28190d4a3888-kube-api-access-2tkt6\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.731944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.731893 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.732021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.731940 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.833880 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.833845 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkt6\" (UniqueName: \"kubernetes.io/projected/947f465c-d8b4-4b83-af9f-28190d4a3888-kube-api-access-2tkt6\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.834035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.833956 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.834035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.833998 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.834691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.834663 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.836641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.836614 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.887441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.887413 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkt6\" (UniqueName: \"kubernetes.io/projected/947f465c-d8b4-4b83-af9f-28190d4a3888-kube-api-access-2tkt6\") pod \"maas-api-revoke-sub-ede02fc0-cjtg5\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:24.938411 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:24.938376 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:25.220427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.220402 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk"] Nov 02 19:48:25.225937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.225912 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.280375 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.280329 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk"] Nov 02 19:48:25.325690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.325642 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5"] Nov 02 19:48:25.330859 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:25.330816 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947f465c_d8b4_4b83_af9f_28190d4a3888.slice/crio-dfd8e50617f343866abd1595ee08fe4e76c56c19bee766cbdde190ec710f9151 WatchSource:0}: Error finding container dfd8e50617f343866abd1595ee08fe4e76c56c19bee766cbdde190ec710f9151: Status 404 returned error can't find the container with id dfd8e50617f343866abd1595ee08fe4e76c56c19bee766cbdde190ec710f9151 Nov 02 19:48:25.342493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.342461 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.343505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.342709 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g82j\" (UniqueName: \"kubernetes.io/projected/1b46edcb-15a4-4da0-b24b-6c5d3309194e-kube-api-access-8g82j\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.343505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.342812 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.445029 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.444993 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.445235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.445212 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.445330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.445264 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g82j\" (UniqueName: \"kubernetes.io/projected/1b46edcb-15a4-4da0-b24b-6c5d3309194e-kube-api-access-8g82j\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.445942 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.445915 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.447571 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.447545 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.498807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.498742 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g82j\" (UniqueName: \"kubernetes.io/projected/1b46edcb-15a4-4da0-b24b-6c5d3309194e-kube-api-access-8g82j\") pod \"maas-api-revoke-sub-cf0d71fd-98gfk\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.544399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.544372 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:25.679095 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.679048 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" event={"ID":"947f465c-d8b4-4b83-af9f-28190d4a3888","Type":"ContainerStarted","Data":"dfd8e50617f343866abd1595ee08fe4e76c56c19bee766cbdde190ec710f9151"} Nov 02 19:48:25.819627 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.819600 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:25.925101 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:25.924908 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b46edcb_15a4_4da0_b24b_6c5d3309194e.slice/crio-9e79f9da61ef8494a6ce28580fc5946ca30192f91b4e56691a3f2290b76f2053 WatchSource:0}: Error finding container 9e79f9da61ef8494a6ce28580fc5946ca30192f91b4e56691a3f2290b76f2053: Status 404 returned error can't find the container with id 9e79f9da61ef8494a6ce28580fc5946ca30192f91b4e56691a3f2290b76f2053 Nov 02 19:48:25.927169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.927148 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk"] Nov 02 19:48:25.951463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.951427 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-xks-ca\") pod \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " Nov 02 19:48:25.951807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.951782 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-openshift-ca\") pod \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " Nov 02 19:48:25.951886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.951826 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-kube-api-access-jglms\" (UniqueName: \"kubernetes.io/projected/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-kube-api-access-jglms\") pod \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\" (UID: \"e5e4ff5c-2443-45aa-afa4-83300ac39fc9\") " Nov 02 19:48:25.952224 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.952188 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-openshift-ca" pod "e5e4ff5c-2443-45aa-afa4-83300ac39fc9" (UID: "e5e4ff5c-2443-45aa-afa4-83300ac39fc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:25.952493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.952471 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:25.954046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.954024 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-xks-ca" pod "e5e4ff5c-2443-45aa-afa4-83300ac39fc9" (UID: "e5e4ff5c-2443-45aa-afa4-83300ac39fc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:25.954141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:25.954069 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-kube-api-access-jglms" pod "e5e4ff5c-2443-45aa-afa4-83300ac39fc9" (UID: "e5e4ff5c-2443-45aa-afa4-83300ac39fc9"). InnerVolumeSpecName "kube-api-access-jglms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:26.053394 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.053325 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jglms\" (UniqueName: \"kubernetes.io/projected/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-kube-api-access-jglms\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:26.053394 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.053352 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4ff5c-2443-45aa-afa4-83300ac39fc9-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:26.692367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.692246 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" Nov 02 19:48:26.692367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.692280 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb" event={"ID":"e5e4ff5c-2443-45aa-afa4-83300ac39fc9","Type":"ContainerDied","Data":"b0ff70417d82db756ef310baff0895e167c351a454ecd3af8b8e9aaaf0d172d8"} Nov 02 19:48:26.692367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.692360 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0ff70417d82db756ef310baff0895e167c351a454ecd3af8b8e9aaaf0d172d8" Nov 02 19:48:26.694803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.694772 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" event={"ID":"1b46edcb-15a4-4da0-b24b-6c5d3309194e","Type":"ContainerStarted","Data":"b0433db0a66eb1ed538a021ed4b1e8573a41884baa562732949466ca07b1efb5"} Nov 02 19:48:26.694927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.694826 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" event={"ID":"1b46edcb-15a4-4da0-b24b-6c5d3309194e","Type":"ContainerStarted","Data":"9e79f9da61ef8494a6ce28580fc5946ca30192f91b4e56691a3f2290b76f2053"} Nov 02 19:48:26.697684 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.697659 4966 generic.go:356] "Generic (PLEG): container finished" podID="947f465c-d8b4-4b83-af9f-28190d4a3888" containerID="a443f0c04c9fb2483206d4dda11b2cb5de7740597c7c79aed6b6d98e059086b9" exitCode=0 Nov 02 19:48:26.697810 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.697743 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" event={"ID":"947f465c-d8b4-4b83-af9f-28190d4a3888","Type":"ContainerDied","Data":"a443f0c04c9fb2483206d4dda11b2cb5de7740597c7c79aed6b6d98e059086b9"} Nov 02 19:48:26.699196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.699174 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx"] Nov 02 19:48:26.699833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.699815 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e5e4ff5c-2443-45aa-afa4-83300ac39fc9" containerName="revoke-keys" Nov 02 19:48:26.699913 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.699834 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e4ff5c-2443-45aa-afa4-83300ac39fc9" containerName="revoke-keys" Nov 02 19:48:26.700024 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.700010 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e4ff5c-2443-45aa-afa4-83300ac39fc9" containerName="revoke-keys" Nov 02 19:48:26.703667 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.703649 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.759346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.758647 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx"] Nov 02 19:48:26.812994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.810895 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" podStartSLOduration=1.8108801319999999 podStartE2EDuration="1.810880132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:26.802206405 +0000 UTC m=+1212.750862245" watchObservedRunningTime="2026-11-02 19:48:26.810880132 +0000 UTC m=+1212.759535971" Nov 02 19:48:26.866746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.866707 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fr9h\" (UniqueName: \"kubernetes.io/projected/26757aed-c515-4053-91ce-bfc01caba2c1-kube-api-access-4fr9h\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.866990 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.866971 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.867131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.867116 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.968828 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.968795 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.968991 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.968962 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fr9h\" (UniqueName: \"kubernetes.io/projected/26757aed-c515-4053-91ce-bfc01caba2c1-kube-api-access-4fr9h\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.969135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.969119 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.969741 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.969711 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:26.971286 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:26.971259 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:27.021817 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.021777 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fr9h\" (UniqueName: \"kubernetes.io/projected/26757aed-c515-4053-91ce-bfc01caba2c1-kube-api-access-4fr9h\") pod \"maas-api-revoke-sub-c948dd00-ww9xx\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:27.032823 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.032792 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:27.428002 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:27.427968 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26757aed_c515_4053_91ce_bfc01caba2c1.slice/crio-f63b4cbd529bc5cae1577467e1492e5db30c1730fa998debb85fb8b34e6fcf07 WatchSource:0}: Error finding container f63b4cbd529bc5cae1577467e1492e5db30c1730fa998debb85fb8b34e6fcf07: Status 404 returned error can't find the container with id f63b4cbd529bc5cae1577467e1492e5db30c1730fa998debb85fb8b34e6fcf07 Nov 02 19:48:27.429196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.429169 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx"] Nov 02 19:48:27.708509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.708419 4966 generic.go:356] "Generic (PLEG): container finished" podID="1b46edcb-15a4-4da0-b24b-6c5d3309194e" containerID="b0433db0a66eb1ed538a021ed4b1e8573a41884baa562732949466ca07b1efb5" exitCode=0 Nov 02 19:48:27.708924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.708499 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" event={"ID":"1b46edcb-15a4-4da0-b24b-6c5d3309194e","Type":"ContainerDied","Data":"b0433db0a66eb1ed538a021ed4b1e8573a41884baa562732949466ca07b1efb5"} Nov 02 19:48:27.709876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.709852 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" event={"ID":"26757aed-c515-4053-91ce-bfc01caba2c1","Type":"ContainerStarted","Data":"f63b4cbd529bc5cae1577467e1492e5db30c1730fa998debb85fb8b34e6fcf07"} Nov 02 19:48:27.853117 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.853087 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:27.986354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.984401 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947f465c-d8b4-4b83-af9f-28190d4a3888-kube-api-access-2tkt6\" (UniqueName: \"kubernetes.io/projected/947f465c-d8b4-4b83-af9f-28190d4a3888-kube-api-access-2tkt6\") pod \"947f465c-d8b4-4b83-af9f-28190d4a3888\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " Nov 02 19:48:27.986354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.984480 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-openshift-ca\") pod \"947f465c-d8b4-4b83-af9f-28190d4a3888\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " Nov 02 19:48:27.986354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.984579 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-xks-ca\") pod \"947f465c-d8b4-4b83-af9f-28190d4a3888\" (UID: \"947f465c-d8b4-4b83-af9f-28190d4a3888\") " Nov 02 19:48:27.986651 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.986369 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-openshift-ca" pod "947f465c-d8b4-4b83-af9f-28190d4a3888" (UID: "947f465c-d8b4-4b83-af9f-28190d4a3888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:27.988714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.988682 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-xks-ca" pod "947f465c-d8b4-4b83-af9f-28190d4a3888" (UID: "947f465c-d8b4-4b83-af9f-28190d4a3888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:27.988856 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:27.988830 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947f465c-d8b4-4b83-af9f-28190d4a3888-kube-api-access-2tkt6" pod "947f465c-d8b4-4b83-af9f-28190d4a3888" (UID: "947f465c-d8b4-4b83-af9f-28190d4a3888"). InnerVolumeSpecName "kube-api-access-2tkt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:28.049048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.049011 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k"] Nov 02 19:48:28.053975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.053944 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528c4cd5-97s5k"] Nov 02 19:48:28.086044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.086015 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tkt6\" (UniqueName: \"kubernetes.io/projected/947f465c-d8b4-4b83-af9f-28190d4a3888-kube-api-access-2tkt6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:28.086044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.086043 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:28.086210 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.086053 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/947f465c-d8b4-4b83-af9f-28190d4a3888-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:28.187801 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.187760 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd0582e-c421-4fa1-8d7b-df8bd79961e1" path="/var/lib/kubelet/pods/3bd0582e-c421-4fa1-8d7b-df8bd79961e1/volumes" Nov 02 19:48:28.718964 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.718941 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" Nov 02 19:48:28.718964 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.718958 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5" event={"ID":"947f465c-d8b4-4b83-af9f-28190d4a3888","Type":"ContainerDied","Data":"dfd8e50617f343866abd1595ee08fe4e76c56c19bee766cbdde190ec710f9151"} Nov 02 19:48:28.719517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.719005 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfd8e50617f343866abd1595ee08fe4e76c56c19bee766cbdde190ec710f9151" Nov 02 19:48:28.721414 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.721372 4966 generic.go:356] "Generic (PLEG): container finished" podID="26757aed-c515-4053-91ce-bfc01caba2c1" containerID="1a1ff91ae136c3c0ea0eb5534737c5342a0bcc0880f8d1f727664a6fbfd8fed1" exitCode=0 Nov 02 19:48:28.722408 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.722358 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" event={"ID":"26757aed-c515-4053-91ce-bfc01caba2c1","Type":"ContainerDied","Data":"1a1ff91ae136c3c0ea0eb5534737c5342a0bcc0880f8d1f727664a6fbfd8fed1"} Nov 02 19:48:28.914639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:28.914613 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:29.099945 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.099857 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b46edcb-15a4-4da0-b24b-6c5d3309194e-kube-api-access-8g82j\" (UniqueName: \"kubernetes.io/projected/1b46edcb-15a4-4da0-b24b-6c5d3309194e-kube-api-access-8g82j\") pod \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " Nov 02 19:48:29.099945 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.099921 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-xks-ca\") pod \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " Nov 02 19:48:29.100183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.100038 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-openshift-ca\") pod \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\" (UID: \"1b46edcb-15a4-4da0-b24b-6c5d3309194e\") " Nov 02 19:48:29.100818 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.100721 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-openshift-ca" pod "1b46edcb-15a4-4da0-b24b-6c5d3309194e" (UID: "1b46edcb-15a4-4da0-b24b-6c5d3309194e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:29.101722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.101695 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:29.115209 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.115167 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b46edcb-15a4-4da0-b24b-6c5d3309194e-kube-api-access-8g82j" pod "1b46edcb-15a4-4da0-b24b-6c5d3309194e" (UID: "1b46edcb-15a4-4da0-b24b-6c5d3309194e"). InnerVolumeSpecName "kube-api-access-8g82j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:29.116437 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.116411 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-xks-ca" pod "1b46edcb-15a4-4da0-b24b-6c5d3309194e" (UID: "1b46edcb-15a4-4da0-b24b-6c5d3309194e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:29.203894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.203853 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g82j\" (UniqueName: \"kubernetes.io/projected/1b46edcb-15a4-4da0-b24b-6c5d3309194e-kube-api-access-8g82j\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:29.203894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.203894 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b46edcb-15a4-4da0-b24b-6c5d3309194e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:29.731073 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.731040 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" Nov 02 19:48:29.731073 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.731052 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk" event={"ID":"1b46edcb-15a4-4da0-b24b-6c5d3309194e","Type":"ContainerDied","Data":"9e79f9da61ef8494a6ce28580fc5946ca30192f91b4e56691a3f2290b76f2053"} Nov 02 19:48:29.731555 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.731108 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e79f9da61ef8494a6ce28580fc5946ca30192f91b4e56691a3f2290b76f2053" Nov 02 19:48:29.901677 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:29.901651 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:30.014809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.014730 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26757aed-c515-4053-91ce-bfc01caba2c1-kube-api-access-4fr9h\" (UniqueName: \"kubernetes.io/projected/26757aed-c515-4053-91ce-bfc01caba2c1-kube-api-access-4fr9h\") pod \"26757aed-c515-4053-91ce-bfc01caba2c1\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " Nov 02 19:48:30.014959 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.014814 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-openshift-ca\") pod \"26757aed-c515-4053-91ce-bfc01caba2c1\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " Nov 02 19:48:30.014959 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.014861 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-xks-ca\") pod \"26757aed-c515-4053-91ce-bfc01caba2c1\" (UID: \"26757aed-c515-4053-91ce-bfc01caba2c1\") " Nov 02 19:48:30.016094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.016066 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-openshift-ca" pod "26757aed-c515-4053-91ce-bfc01caba2c1" (UID: "26757aed-c515-4053-91ce-bfc01caba2c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:30.017823 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.017788 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26757aed-c515-4053-91ce-bfc01caba2c1-kube-api-access-4fr9h" pod "26757aed-c515-4053-91ce-bfc01caba2c1" (UID: "26757aed-c515-4053-91ce-bfc01caba2c1"). InnerVolumeSpecName "kube-api-access-4fr9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:30.018437 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.018410 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-xks-ca" pod "26757aed-c515-4053-91ce-bfc01caba2c1" (UID: "26757aed-c515-4053-91ce-bfc01caba2c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:30.116509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.116465 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fr9h\" (UniqueName: \"kubernetes.io/projected/26757aed-c515-4053-91ce-bfc01caba2c1-kube-api-access-4fr9h\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:30.116509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.116501 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:30.116710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.116518 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26757aed-c515-4053-91ce-bfc01caba2c1-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:30.741685 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.741653 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" Nov 02 19:48:30.742091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.741683 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx" event={"ID":"26757aed-c515-4053-91ce-bfc01caba2c1","Type":"ContainerDied","Data":"f63b4cbd529bc5cae1577467e1492e5db30c1730fa998debb85fb8b34e6fcf07"} Nov 02 19:48:30.742091 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:30.741744 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f63b4cbd529bc5cae1577467e1492e5db30c1730fa998debb85fb8b34e6fcf07" Nov 02 19:48:33.051710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:33.051667 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp"] Nov 02 19:48:33.096854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:33.096817 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq"] Nov 02 19:48:33.102716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:33.102681 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30926ff4-qjzwp"] Nov 02 19:48:33.106646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:33.106620 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c045110-zjrtq"] Nov 02 19:48:34.183788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:34.183756 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001f1027-384c-4afe-b6c7-119968977c98" path="/var/lib/kubelet/pods/001f1027-384c-4afe-b6c7-119968977c98/volumes" Nov 02 19:48:34.184240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:34.184226 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4f2c2e-e36d-449f-99dd-029fd965d56a" path="/var/lib/kubelet/pods/dd4f2c2e-e36d-449f-99dd-029fd965d56a/volumes" Nov 02 19:48:36.197534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.197499 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b"] Nov 02 19:48:36.198445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198424 4966 cpu_manager.go:414] "containerMap: removing container" podUID="26757aed-c515-4053-91ce-bfc01caba2c1" containerName="revoke-keys" Nov 02 19:48:36.198500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198448 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26757aed-c515-4053-91ce-bfc01caba2c1" containerName="revoke-keys" Nov 02 19:48:36.198500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198465 4966 cpu_manager.go:414] "containerMap: removing container" podUID="947f465c-d8b4-4b83-af9f-28190d4a3888" containerName="revoke-keys" Nov 02 19:48:36.198500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198474 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947f465c-d8b4-4b83-af9f-28190d4a3888" containerName="revoke-keys" Nov 02 19:48:36.198500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198490 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1b46edcb-15a4-4da0-b24b-6c5d3309194e" containerName="revoke-keys" Nov 02 19:48:36.198500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198498 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b46edcb-15a4-4da0-b24b-6c5d3309194e" containerName="revoke-keys" Nov 02 19:48:36.198660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198648 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="26757aed-c515-4053-91ce-bfc01caba2c1" containerName="revoke-keys" Nov 02 19:48:36.198694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198666 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="947f465c-d8b4-4b83-af9f-28190d4a3888" containerName="revoke-keys" Nov 02 19:48:36.198694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.198678 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b46edcb-15a4-4da0-b24b-6c5d3309194e" containerName="revoke-keys" Nov 02 19:48:36.203831 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.203809 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.253077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.253044 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b"] Nov 02 19:48:36.293337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.293279 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:48:36.293499 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.293366 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:48:36.297332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.297282 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfbf8\" (UniqueName: \"kubernetes.io/projected/78a4bf4e-872e-4e49-8d7c-a1f418210eef-kube-api-access-gfbf8\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.297491 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.297446 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.297491 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.297485 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.298112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.298085 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:48:36.298248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.298116 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:48:36.398751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.398720 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfbf8\" (UniqueName: \"kubernetes.io/projected/78a4bf4e-872e-4e49-8d7c-a1f418210eef-kube-api-access-gfbf8\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.398926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.398798 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.398926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.398829 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.399451 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.399431 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.401134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.401105 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.454188 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.454130 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfbf8\" (UniqueName: \"kubernetes.io/projected/78a4bf4e-872e-4e49-8d7c-a1f418210eef-kube-api-access-gfbf8\") pod \"maas-api-revoke-sub-9eaef11f-7sq6b\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.520228 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.520196 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:36.895507 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:36.895471 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a4bf4e_872e_4e49_8d7c_a1f418210eef.slice/crio-f61f700a4dd9734c6e5a01c2fb64ce359384d5bb1a7463f211198f8a0f025ecb WatchSource:0}: Error finding container f61f700a4dd9734c6e5a01c2fb64ce359384d5bb1a7463f211198f8a0f025ecb: Status 404 returned error can't find the container with id f61f700a4dd9734c6e5a01c2fb64ce359384d5bb1a7463f211198f8a0f025ecb Nov 02 19:48:36.896785 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:36.896757 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b"] Nov 02 19:48:37.805715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:37.805677 4966 generic.go:356] "Generic (PLEG): container finished" podID="78a4bf4e-872e-4e49-8d7c-a1f418210eef" containerID="8baa1ab625b11e025ab83849d762c1ed5e5819673adb4ccf17e59e57b8641fb6" exitCode=0 Nov 02 19:48:37.806393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:37.806101 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" event={"ID":"78a4bf4e-872e-4e49-8d7c-a1f418210eef","Type":"ContainerDied","Data":"8baa1ab625b11e025ab83849d762c1ed5e5819673adb4ccf17e59e57b8641fb6"} Nov 02 19:48:37.806393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:37.806223 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" event={"ID":"78a4bf4e-872e-4e49-8d7c-a1f418210eef","Type":"ContainerStarted","Data":"f61f700a4dd9734c6e5a01c2fb64ce359384d5bb1a7463f211198f8a0f025ecb"} Nov 02 19:48:38.453135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.453100 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb"] Nov 02 19:48:38.457888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.457864 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.511443 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.511403 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb"] Nov 02 19:48:38.528164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.528131 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.528164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.528167 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx5k4\" (UniqueName: \"kubernetes.io/projected/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-kube-api-access-vx5k4\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.528461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.528189 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.629832 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.629790 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.630009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.629837 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx5k4\" (UniqueName: \"kubernetes.io/projected/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-kube-api-access-vx5k4\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.630009 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.629876 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.630647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.630621 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.632618 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.632590 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.683159 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.683122 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx5k4\" (UniqueName: \"kubernetes.io/projected/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-kube-api-access-vx5k4\") pod \"maas-api-revoke-sub-108cbcef-6pscb\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:38.772738 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:38.772704 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:39.025197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.024807 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:39.136818 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.136784 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-xks-ca\") pod \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " Nov 02 19:48:39.137027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.136831 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-openshift-ca\") pod \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " Nov 02 19:48:39.137027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.136905 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a4bf4e-872e-4e49-8d7c-a1f418210eef-kube-api-access-gfbf8\" (UniqueName: \"kubernetes.io/projected/78a4bf4e-872e-4e49-8d7c-a1f418210eef-kube-api-access-gfbf8\") pod \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\" (UID: \"78a4bf4e-872e-4e49-8d7c-a1f418210eef\") " Nov 02 19:48:39.137727 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.137690 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-openshift-ca" pod "78a4bf4e-872e-4e49-8d7c-a1f418210eef" (UID: "78a4bf4e-872e-4e49-8d7c-a1f418210eef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:39.139515 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.139485 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a4bf4e-872e-4e49-8d7c-a1f418210eef-kube-api-access-gfbf8" pod "78a4bf4e-872e-4e49-8d7c-a1f418210eef" (UID: "78a4bf4e-872e-4e49-8d7c-a1f418210eef"). InnerVolumeSpecName "kube-api-access-gfbf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:39.140211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.140186 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-xks-ca" pod "78a4bf4e-872e-4e49-8d7c-a1f418210eef" (UID: "78a4bf4e-872e-4e49-8d7c-a1f418210eef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:39.157332 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:39.157286 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e92616f_1c77_4ad5_9f8d_12f61183fa1c.slice/crio-b137c38a54f3ad0e51f943ea2861831f14b75db8bba54b0680acc80dfbb3480f WatchSource:0}: Error finding container b137c38a54f3ad0e51f943ea2861831f14b75db8bba54b0680acc80dfbb3480f: Status 404 returned error can't find the container with id b137c38a54f3ad0e51f943ea2861831f14b75db8bba54b0680acc80dfbb3480f Nov 02 19:48:39.185621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.185592 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb"] Nov 02 19:48:39.238662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.238623 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfbf8\" (UniqueName: \"kubernetes.io/projected/78a4bf4e-872e-4e49-8d7c-a1f418210eef-kube-api-access-gfbf8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:39.238662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.238656 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:39.238837 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.238670 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a4bf4e-872e-4e49-8d7c-a1f418210eef-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:39.825430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.825390 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" event={"ID":"7e92616f-1c77-4ad5-9f8d-12f61183fa1c","Type":"ContainerStarted","Data":"714b400a92a926f640f260437b12c5ad13da7aaae409e801f85b0a3f696d4d99"} Nov 02 19:48:39.825639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.825449 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" event={"ID":"7e92616f-1c77-4ad5-9f8d-12f61183fa1c","Type":"ContainerStarted","Data":"b137c38a54f3ad0e51f943ea2861831f14b75db8bba54b0680acc80dfbb3480f"} Nov 02 19:48:39.826960 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.826936 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" Nov 02 19:48:39.827093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.826985 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b" event={"ID":"78a4bf4e-872e-4e49-8d7c-a1f418210eef","Type":"ContainerDied","Data":"f61f700a4dd9734c6e5a01c2fb64ce359384d5bb1a7463f211198f8a0f025ecb"} Nov 02 19:48:39.827093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.827035 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f61f700a4dd9734c6e5a01c2fb64ce359384d5bb1a7463f211198f8a0f025ecb" Nov 02 19:48:39.932672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:39.932573 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" podStartSLOduration=1.932558252 podStartE2EDuration="1.932558252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:39.930649651 +0000 UTC m=+1225.879305473" watchObservedRunningTime="2026-11-02 19:48:39.932558252 +0000 UTC m=+1225.881214092" Nov 02 19:48:40.837592 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:40.837555 4966 generic.go:356] "Generic (PLEG): container finished" podID="7e92616f-1c77-4ad5-9f8d-12f61183fa1c" containerID="714b400a92a926f640f260437b12c5ad13da7aaae409e801f85b0a3f696d4d99" exitCode=0 Nov 02 19:48:40.838023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:40.837648 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" event={"ID":"7e92616f-1c77-4ad5-9f8d-12f61183fa1c","Type":"ContainerDied","Data":"714b400a92a926f640f260437b12c5ad13da7aaae409e801f85b0a3f696d4d99"} Nov 02 19:48:41.071993 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:41.071953 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc"] Nov 02 19:48:41.076513 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:41.076485 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dbafe-zcnbc"] Nov 02 19:48:42.033190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.033163 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:42.177144 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.177051 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-openshift-ca\") pod \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " Nov 02 19:48:42.177340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.177148 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-xks-ca\") pod \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " Nov 02 19:48:42.177340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.177244 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-kube-api-access-vx5k4\" (UniqueName: \"kubernetes.io/projected/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-kube-api-access-vx5k4\") pod \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\" (UID: \"7e92616f-1c77-4ad5-9f8d-12f61183fa1c\") " Nov 02 19:48:42.177547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.177519 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-openshift-ca" pod "7e92616f-1c77-4ad5-9f8d-12f61183fa1c" (UID: "7e92616f-1c77-4ad5-9f8d-12f61183fa1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:42.177979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.177952 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:42.179611 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.179580 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-kube-api-access-vx5k4" pod "7e92616f-1c77-4ad5-9f8d-12f61183fa1c" (UID: "7e92616f-1c77-4ad5-9f8d-12f61183fa1c"). InnerVolumeSpecName "kube-api-access-vx5k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:42.179857 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.179825 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-xks-ca" pod "7e92616f-1c77-4ad5-9f8d-12f61183fa1c" (UID: "7e92616f-1c77-4ad5-9f8d-12f61183fa1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:42.184819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.184795 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="466eced8-8d4f-4691-9b43-e96e7af148e5" path="/var/lib/kubelet/pods/466eced8-8d4f-4691-9b43-e96e7af148e5/volumes" Nov 02 19:48:42.280276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.280230 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:42.280276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.280269 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx5k4\" (UniqueName: \"kubernetes.io/projected/7e92616f-1c77-4ad5-9f8d-12f61183fa1c-kube-api-access-vx5k4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:42.860696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.860666 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" Nov 02 19:48:42.860696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.860677 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb" event={"ID":"7e92616f-1c77-4ad5-9f8d-12f61183fa1c","Type":"ContainerDied","Data":"b137c38a54f3ad0e51f943ea2861831f14b75db8bba54b0680acc80dfbb3480f"} Nov 02 19:48:42.860938 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:42.860731 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b137c38a54f3ad0e51f943ea2861831f14b75db8bba54b0680acc80dfbb3480f" Nov 02 19:48:44.048402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:44.048364 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98"] Nov 02 19:48:44.091979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:44.091942 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2822e1b-kzl98"] Nov 02 19:48:44.186787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:44.186749 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c48a06-a93d-48a7-9b8a-c80e59ba650b" path="/var/lib/kubelet/pods/f9c48a06-a93d-48a7-9b8a-c80e59ba650b/volumes" Nov 02 19:48:44.753648 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:48:44.753601 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:48:48.045251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:48.045214 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh"] Nov 02 19:48:48.091570 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:48.091533 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79202cf8-tvqvh"] Nov 02 19:48:48.184211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:48.184178 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd50657-fcc9-44f0-b811-72a952a22fb8" path="/var/lib/kubelet/pods/ebd50657-fcc9-44f0-b811-72a952a22fb8/volumes" Nov 02 19:48:49.046902 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.046864 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt"] Nov 02 19:48:49.092789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.092752 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3744a939-x6snt"] Nov 02 19:48:49.849599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.849566 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj"] Nov 02 19:48:49.850870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.850824 4966 cpu_manager.go:414] "containerMap: removing container" podUID="78a4bf4e-872e-4e49-8d7c-a1f418210eef" containerName="revoke-keys" Nov 02 19:48:49.851030 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.851016 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a4bf4e-872e-4e49-8d7c-a1f418210eef" containerName="revoke-keys" Nov 02 19:48:49.851139 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.851130 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7e92616f-1c77-4ad5-9f8d-12f61183fa1c" containerName="revoke-keys" Nov 02 19:48:49.851218 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.851209 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e92616f-1c77-4ad5-9f8d-12f61183fa1c" containerName="revoke-keys" Nov 02 19:48:49.851478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.851464 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a4bf4e-872e-4e49-8d7c-a1f418210eef" containerName="revoke-keys" Nov 02 19:48:49.851585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.851575 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e92616f-1c77-4ad5-9f8d-12f61183fa1c" containerName="revoke-keys" Nov 02 19:48:49.863501 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:49.863473 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227156 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227240 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227349 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b4d281-badc-4056-8530-1117729fde32-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227432 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227470 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227496 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvpg9\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-kube-api-access-bvpg9\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227590 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227649 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.229327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.227733 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b4d281-badc-4056-8530-1117729fde32-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369093 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b4d281-badc-4056-8530-1117729fde32-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369185 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369232 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369323 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b4d281-badc-4056-8530-1117729fde32-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369398 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369432 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369458 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpg9\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-kube-api-access-bvpg9\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369547 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.370321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.369602 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.416006 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.416345 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.416621 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.417312 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b4d281-badc-4056-8530-1117729fde32-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.422736 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.434712 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.441610 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.475502 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpg9\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-kube-api-access-bvpg9\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.487759 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93" path="/var/lib/kubelet/pods/d2da2b11-aa7f-45b5-b4dc-0ff279c1ba93/volumes" Nov 02 19:48:50.505578 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.488231 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj"] Nov 02 19:48:50.536368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.527070 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b4d281-badc-4056-8530-1117729fde32-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:50.838678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:50.838590 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:51.397322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.396427 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.437468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.437415 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.446813 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.446768 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:48:51.463666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.463607 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.479497 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.479446 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.525713 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.525676 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.583364 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.571752 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 19:48:51.601120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.599403 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.656386 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.646391 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:51.716944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.716904 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Nov 02 19:48:51.920546 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:51.920418 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:48:52.073323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.070094 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj"] Nov 02 19:48:52.078753 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:52.078697 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b4d281_badc_4056_8530_1117729fde32.slice/crio-96567790ad55d7a63b0f1ab4ba5f9450852d30e63be1f78a41227f78f0942434 WatchSource:0}: Error finding container 96567790ad55d7a63b0f1ab4ba5f9450852d30e63be1f78a41227f78f0942434: Status 404 returned error can't find the container with id 96567790ad55d7a63b0f1ab4ba5f9450852d30e63be1f78a41227f78f0942434 Nov 02 19:48:52.084399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.084353 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:52.084689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.084657 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:52.090506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.090473 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:48:52.649184 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.649134 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" event={"ID":"e7b4d281-badc-4056-8530-1117729fde32","Type":"ContainerStarted","Data":"5a6742d41faa265bd077d802dde3ffc7a97e9ecbc0abd03902e0f192b145a3ca"} Nov 02 19:48:52.649625 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.649214 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" event={"ID":"e7b4d281-badc-4056-8530-1117729fde32","Type":"ContainerStarted","Data":"96567790ad55d7a63b0f1ab4ba5f9450852d30e63be1f78a41227f78f0942434"} Nov 02 19:48:52.796343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.793733 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podStartSLOduration=3.793710787 podStartE2EDuration="3.793710787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:52.774938021 +0000 UTC m=+1238.723593890" watchObservedRunningTime="2026-11-02 19:48:52.793710787 +0000 UTC m=+1238.742366629" Nov 02 19:48:52.839606 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:52.839566 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:53.844425 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:53.844387 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 19:48:53.845021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:53.844988 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 19:48:54.708455 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.708421 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m"] Nov 02 19:48:54.713016 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.712992 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.754331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.753089 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.754331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.753266 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.754331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.753542 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkv2\" (UniqueName: \"kubernetes.io/projected/8c6ef62a-982c-47d9-85b0-c1f4873013a8-kube-api-access-hkkv2\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.776627 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.776595 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m"] Nov 02 19:48:54.810686 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.810657 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:48:54.811037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.810669 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:48:54.811037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.810799 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:48:54.811230 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.811024 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:48:54.865332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.862669 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 19:48:54.865332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.862729 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 19:48:54.865332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.863213 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.865332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.863409 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkv2\" (UniqueName: \"kubernetes.io/projected/8c6ef62a-982c-47d9-85b0-c1f4873013a8-kube-api-access-hkkv2\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.865332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.863498 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.869324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.868061 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.871084 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.870533 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:54.921429 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:54.921396 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkv2\" (UniqueName: \"kubernetes.io/projected/8c6ef62a-982c-47d9-85b0-c1f4873013a8-kube-api-access-hkkv2\") pod \"maas-api-revoke-sub-341acaec-nqp6m\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:55.027748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.027716 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:55.071111 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.071064 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6"] Nov 02 19:48:55.083568 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.083527 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17bfcd5e-n6pl6"] Nov 02 19:48:55.469046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.468587 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m"] Nov 02 19:48:55.470842 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:55.470804 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6ef62a_982c_47d9_85b0_c1f4873013a8.slice/crio-584d11c31e741b5f53e3c8b26104eac44c23ec5ab0e825350e9852c2f4846617 WatchSource:0}: Error finding container 584d11c31e741b5f53e3c8b26104eac44c23ec5ab0e825350e9852c2f4846617: Status 404 returned error can't find the container with id 584d11c31e741b5f53e3c8b26104eac44c23ec5ab0e825350e9852c2f4846617 Nov 02 19:48:55.686699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.686669 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8"] Nov 02 19:48:55.692254 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.692234 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.702016 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.701991 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" event={"ID":"8c6ef62a-982c-47d9-85b0-c1f4873013a8","Type":"ContainerStarted","Data":"584d11c31e741b5f53e3c8b26104eac44c23ec5ab0e825350e9852c2f4846617"} Nov 02 19:48:55.778642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.778551 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8"] Nov 02 19:48:55.787239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.787193 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.787448 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.787426 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.787556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.787524 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dxbd\" (UniqueName: \"kubernetes.io/projected/8536d875-ad0d-4a5c-bdb0-55720314e10e-kube-api-access-2dxbd\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.841772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.841735 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 19:48:55.861326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.842021 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 19:48:55.888677 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.888645 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.889100 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.888772 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.889100 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.888794 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dxbd\" (UniqueName: \"kubernetes.io/projected/8536d875-ad0d-4a5c-bdb0-55720314e10e-kube-api-access-2dxbd\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.889488 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.889461 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.891222 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.891202 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:55.944006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:55.943977 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dxbd\" (UniqueName: \"kubernetes.io/projected/8536d875-ad0d-4a5c-bdb0-55720314e10e-kube-api-access-2dxbd\") pod \"maas-api-revoke-sub-78d5981b-dh9f8\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:56.007364 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.007332 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:56.187885 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.187851 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4185f19e-e5fb-4092-9f86-de99ea5d6304" path="/var/lib/kubelet/pods/4185f19e-e5fb-4092-9f86-de99ea5d6304/volumes" Nov 02 19:48:56.414126 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:56.414096 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8536d875_ad0d_4a5c_bdb0_55720314e10e.slice/crio-4f6475e41957fafc7dbf6b43082f5494a23eb449660e5b80e721f604e1c95103 WatchSource:0}: Error finding container 4f6475e41957fafc7dbf6b43082f5494a23eb449660e5b80e721f604e1c95103: Status 404 returned error can't find the container with id 4f6475e41957fafc7dbf6b43082f5494a23eb449660e5b80e721f604e1c95103 Nov 02 19:48:56.414252 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.414144 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8"] Nov 02 19:48:56.711872 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.711838 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" event={"ID":"8536d875-ad0d-4a5c-bdb0-55720314e10e","Type":"ContainerStarted","Data":"4f6475e41957fafc7dbf6b43082f5494a23eb449660e5b80e721f604e1c95103"} Nov 02 19:48:56.713768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.713734 4966 generic.go:356] "Generic (PLEG): container finished" podID="8c6ef62a-982c-47d9-85b0-c1f4873013a8" containerID="b40946ec6d6437ef7fee647a904edc10169c68a6a73f9c3bdd367f6ca47a70f5" exitCode=0 Nov 02 19:48:56.713905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.713781 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" event={"ID":"8c6ef62a-982c-47d9-85b0-c1f4873013a8","Type":"ContainerDied","Data":"b40946ec6d6437ef7fee647a904edc10169c68a6a73f9c3bdd367f6ca47a70f5"} Nov 02 19:48:56.725284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.725260 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq"] Nov 02 19:48:56.729606 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.729589 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.784010 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.783377 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq"] Nov 02 19:48:56.803588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.803557 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.803752 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.803594 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bdj\" (UniqueName: \"kubernetes.io/projected/60e548f9-d865-4521-827a-e2168ed336c7-kube-api-access-f6bdj\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.803752 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.803703 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.862896 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.862852 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 19:48:56.863080 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.862911 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 19:48:56.905283 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.905234 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.905712 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.905309 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bdj\" (UniqueName: \"kubernetes.io/projected/60e548f9-d865-4521-827a-e2168ed336c7-kube-api-access-f6bdj\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.905712 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.905358 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.906096 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.906075 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.907978 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.907958 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:56.967496 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:56.967412 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bdj\" (UniqueName: \"kubernetes.io/projected/60e548f9-d865-4521-827a-e2168ed336c7-kube-api-access-f6bdj\") pod \"maas-api-revoke-sub-fe3ac97c-5mrmq\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:57.066765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:57.066711 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:48:57.476372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:57.476317 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq"] Nov 02 19:48:57.493344 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:48:57.493310 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e548f9_d865_4521_827a_e2168ed336c7.slice/crio-3bc21b379727d4fe8c9235efcc9dfbd0390f916f8ec8362c63f5c090c0bf5113 WatchSource:0}: Error finding container 3bc21b379727d4fe8c9235efcc9dfbd0390f916f8ec8362c63f5c090c0bf5113: Status 404 returned error can't find the container with id 3bc21b379727d4fe8c9235efcc9dfbd0390f916f8ec8362c63f5c090c0bf5113 Nov 02 19:48:57.499722 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:48:57.499695 4966 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8536d875_ad0d_4a5c_bdb0_55720314e10e.slice/crio-conmon-bb5ca403e9d5ef9783fc422e75b83d62a79a01d3ace5db9e9143354e9cbc3621.scope\": RecentStats: unable to find data in memory cache]" Nov 02 19:48:57.725675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:57.725644 4966 generic.go:356] "Generic (PLEG): container finished" podID="8536d875-ad0d-4a5c-bdb0-55720314e10e" containerID="bb5ca403e9d5ef9783fc422e75b83d62a79a01d3ace5db9e9143354e9cbc3621" exitCode=0 Nov 02 19:48:57.725825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:57.725725 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" event={"ID":"8536d875-ad0d-4a5c-bdb0-55720314e10e","Type":"ContainerDied","Data":"bb5ca403e9d5ef9783fc422e75b83d62a79a01d3ace5db9e9143354e9cbc3621"} Nov 02 19:48:57.727281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:57.727254 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" event={"ID":"60e548f9-d865-4521-827a-e2168ed336c7","Type":"ContainerStarted","Data":"3bc21b379727d4fe8c9235efcc9dfbd0390f916f8ec8362c63f5c090c0bf5113"} Nov 02 19:48:57.843196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:57.842724 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 19:48:57.843196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:57.842788 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 19:48:58.004660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.004637 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:58.124873 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.124840 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6ef62a-982c-47d9-85b0-c1f4873013a8-kube-api-access-hkkv2\" (UniqueName: \"kubernetes.io/projected/8c6ef62a-982c-47d9-85b0-c1f4873013a8-kube-api-access-hkkv2\") pod \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " Nov 02 19:48:58.125051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.124946 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-openshift-ca\") pod \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " Nov 02 19:48:58.125412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.125383 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-openshift-ca" pod "8c6ef62a-982c-47d9-85b0-c1f4873013a8" (UID: "8c6ef62a-982c-47d9-85b0-c1f4873013a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:58.125654 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.125632 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-xks-ca\") pod \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\" (UID: \"8c6ef62a-982c-47d9-85b0-c1f4873013a8\") " Nov 02 19:48:58.126217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.126192 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:58.127599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.127569 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6ef62a-982c-47d9-85b0-c1f4873013a8-kube-api-access-hkkv2" pod "8c6ef62a-982c-47d9-85b0-c1f4873013a8" (UID: "8c6ef62a-982c-47d9-85b0-c1f4873013a8"). InnerVolumeSpecName "kube-api-access-hkkv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:58.128522 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.128501 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-xks-ca" pod "8c6ef62a-982c-47d9-85b0-c1f4873013a8" (UID: "8c6ef62a-982c-47d9-85b0-c1f4873013a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:58.228509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.228459 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c6ef62a-982c-47d9-85b0-c1f4873013a8-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:58.228509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.228487 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkkv2\" (UniqueName: \"kubernetes.io/projected/8c6ef62a-982c-47d9-85b0-c1f4873013a8-kube-api-access-hkkv2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:58.739132 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.739088 4966 generic.go:356] "Generic (PLEG): container finished" podID="60e548f9-d865-4521-827a-e2168ed336c7" containerID="617ce219a29f3265e5411530c0e2aa328f5b1802ec5b36ccc9c325ab103a0c52" exitCode=0 Nov 02 19:48:58.739401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.739179 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" event={"ID":"60e548f9-d865-4521-827a-e2168ed336c7","Type":"ContainerDied","Data":"617ce219a29f3265e5411530c0e2aa328f5b1802ec5b36ccc9c325ab103a0c52"} Nov 02 19:48:58.741033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.741009 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" Nov 02 19:48:58.741158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.741032 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m" event={"ID":"8c6ef62a-982c-47d9-85b0-c1f4873013a8","Type":"ContainerDied","Data":"584d11c31e741b5f53e3c8b26104eac44c23ec5ab0e825350e9852c2f4846617"} Nov 02 19:48:58.741158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.741077 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="584d11c31e741b5f53e3c8b26104eac44c23ec5ab0e825350e9852c2f4846617" Nov 02 19:48:58.840343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.840231 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 19:48:58.840744 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.840631 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 19:48:58.918566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:58.918540 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:59.047916 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.047875 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-openshift-ca\") pod \"8536d875-ad0d-4a5c-bdb0-55720314e10e\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " Nov 02 19:48:59.048412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.047992 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8536d875-ad0d-4a5c-bdb0-55720314e10e-kube-api-access-2dxbd\" (UniqueName: \"kubernetes.io/projected/8536d875-ad0d-4a5c-bdb0-55720314e10e-kube-api-access-2dxbd\") pod \"8536d875-ad0d-4a5c-bdb0-55720314e10e\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " Nov 02 19:48:59.048412 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.048148 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-xks-ca\") pod \"8536d875-ad0d-4a5c-bdb0-55720314e10e\" (UID: \"8536d875-ad0d-4a5c-bdb0-55720314e10e\") " Nov 02 19:48:59.049017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.048987 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-openshift-ca" pod "8536d875-ad0d-4a5c-bdb0-55720314e10e" (UID: "8536d875-ad0d-4a5c-bdb0-55720314e10e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:59.051485 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.051456 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8536d875-ad0d-4a5c-bdb0-55720314e10e-kube-api-access-2dxbd" pod "8536d875-ad0d-4a5c-bdb0-55720314e10e" (UID: "8536d875-ad0d-4a5c-bdb0-55720314e10e"). InnerVolumeSpecName "kube-api-access-2dxbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:59.051579 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.051516 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-xks-ca" pod "8536d875-ad0d-4a5c-bdb0-55720314e10e" (UID: "8536d875-ad0d-4a5c-bdb0-55720314e10e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:59.149986 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.149904 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:59.149986 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.149937 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dxbd\" (UniqueName: \"kubernetes.io/projected/8536d875-ad0d-4a5c-bdb0-55720314e10e-kube-api-access-2dxbd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:59.149986 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.149949 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8536d875-ad0d-4a5c-bdb0-55720314e10e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:59.750931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.750905 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" Nov 02 19:48:59.751187 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.750948 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8" event={"ID":"8536d875-ad0d-4a5c-bdb0-55720314e10e","Type":"ContainerDied","Data":"4f6475e41957fafc7dbf6b43082f5494a23eb449660e5b80e721f604e1c95103"} Nov 02 19:48:59.751355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.751205 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f6475e41957fafc7dbf6b43082f5494a23eb449660e5b80e721f604e1c95103" Nov 02 19:48:59.847089 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.845611 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:59.847089 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.846934 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:59.848114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.847767 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:48:59.935614 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:48:59.935591 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:49:00.047020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.046933 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2"] Nov 02 19:49:00.051505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.051477 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f"] Nov 02 19:49:00.065594 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.065553 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-openshift-ca\") pod \"60e548f9-d865-4521-827a-e2168ed336c7\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " Nov 02 19:49:00.065808 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.065775 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-xks-ca\") pod \"60e548f9-d865-4521-827a-e2168ed336c7\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " Nov 02 19:49:00.065957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.065939 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e548f9-d865-4521-827a-e2168ed336c7-kube-api-access-f6bdj\" (UniqueName: \"kubernetes.io/projected/60e548f9-d865-4521-827a-e2168ed336c7-kube-api-access-f6bdj\") pod \"60e548f9-d865-4521-827a-e2168ed336c7\" (UID: \"60e548f9-d865-4521-827a-e2168ed336c7\") " Nov 02 19:49:00.067321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.067233 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-openshift-ca" pod "60e548f9-d865-4521-827a-e2168ed336c7" (UID: "60e548f9-d865-4521-827a-e2168ed336c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:00.069235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.069203 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e548f9-d865-4521-827a-e2168ed336c7-kube-api-access-f6bdj" pod "60e548f9-d865-4521-827a-e2168ed336c7" (UID: "60e548f9-d865-4521-827a-e2168ed336c7"). InnerVolumeSpecName "kube-api-access-f6bdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:00.069404 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.069282 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-xks-ca" pod "60e548f9-d865-4521-827a-e2168ed336c7" (UID: "60e548f9-d865-4521-827a-e2168ed336c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:00.072090 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.072067 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5178f22d-m5zz2"] Nov 02 19:49:00.076921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.076899 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a5adb8-bkh6f"] Nov 02 19:49:00.167872 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.167827 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:00.168032 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.167904 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bdj\" (UniqueName: \"kubernetes.io/projected/60e548f9-d865-4521-827a-e2168ed336c7-kube-api-access-f6bdj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:00.168032 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.167922 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e548f9-d865-4521-827a-e2168ed336c7-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:00.190268 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.190232 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbd6486-9d1a-422c-b84a-0ac2b8c99cba" path="/var/lib/kubelet/pods/9dbd6486-9d1a-422c-b84a-0ac2b8c99cba/volumes" Nov 02 19:49:00.190839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.190816 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9e08b0-4a06-4872-8591-9c965f2261d5" path="/var/lib/kubelet/pods/cb9e08b0-4a06-4872-8591-9c965f2261d5/volumes" Nov 02 19:49:00.766243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.766210 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" Nov 02 19:49:00.767478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.766247 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq" event={"ID":"60e548f9-d865-4521-827a-e2168ed336c7","Type":"ContainerDied","Data":"3bc21b379727d4fe8c9235efcc9dfbd0390f916f8ec8362c63f5c090c0bf5113"} Nov 02 19:49:00.767682 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:00.767658 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc21b379727d4fe8c9235efcc9dfbd0390f916f8ec8362c63f5c090c0bf5113" Nov 02 19:49:02.571723 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.571690 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks"] Nov 02 19:49:02.572355 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572338 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8c6ef62a-982c-47d9-85b0-c1f4873013a8" containerName="revoke-keys" Nov 02 19:49:02.572427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572356 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6ef62a-982c-47d9-85b0-c1f4873013a8" containerName="revoke-keys" Nov 02 19:49:02.572427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572396 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8536d875-ad0d-4a5c-bdb0-55720314e10e" containerName="revoke-keys" Nov 02 19:49:02.572427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572403 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8536d875-ad0d-4a5c-bdb0-55720314e10e" containerName="revoke-keys" Nov 02 19:49:02.572427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572418 4966 cpu_manager.go:414] "containerMap: removing container" podUID="60e548f9-d865-4521-827a-e2168ed336c7" containerName="revoke-keys" Nov 02 19:49:02.572427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572423 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e548f9-d865-4521-827a-e2168ed336c7" containerName="revoke-keys" Nov 02 19:49:02.572573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572523 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6ef62a-982c-47d9-85b0-c1f4873013a8" containerName="revoke-keys" Nov 02 19:49:02.572573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572538 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8536d875-ad0d-4a5c-bdb0-55720314e10e" containerName="revoke-keys" Nov 02 19:49:02.572573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.572546 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e548f9-d865-4521-827a-e2168ed336c7" containerName="revoke-keys" Nov 02 19:49:02.576108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.576091 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.629443 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.629413 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks"] Nov 02 19:49:02.669981 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.669943 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:49:02.671083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.671062 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:49:02.671083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.671075 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:49:02.671271 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.671079 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:49:02.704769 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.704737 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.704910 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.704859 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh2x\" (UniqueName: \"kubernetes.io/projected/67118bc7-f99c-47f8-894d-8d82423e1435-kube-api-access-fmh2x\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.704977 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.704906 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.805769 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.805728 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.805968 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.805840 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh2x\" (UniqueName: \"kubernetes.io/projected/67118bc7-f99c-47f8-894d-8d82423e1435-kube-api-access-fmh2x\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.805968 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.805887 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.806574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.806556 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.808122 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.808106 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.859000 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.858931 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh2x\" (UniqueName: \"kubernetes.io/projected/67118bc7-f99c-47f8-894d-8d82423e1435-kube-api-access-fmh2x\") pod \"maas-api-revoke-sub-25fdc4fd-64cks\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:02.891398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:02.891368 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:03.267336 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:03.266335 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67118bc7_f99c_47f8_894d_8d82423e1435.slice/crio-c52a875b5397dedf7893eaac9a6d3e8d0d09d423bc6cb2d1b644b6cf34c6bd75 WatchSource:0}: Error finding container c52a875b5397dedf7893eaac9a6d3e8d0d09d423bc6cb2d1b644b6cf34c6bd75: Status 404 returned error can't find the container with id c52a875b5397dedf7893eaac9a6d3e8d0d09d423bc6cb2d1b644b6cf34c6bd75 Nov 02 19:49:03.267336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:03.266367 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks"] Nov 02 19:49:03.808131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:03.808019 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" event={"ID":"67118bc7-f99c-47f8-894d-8d82423e1435","Type":"ContainerStarted","Data":"b77206b6a3a0f5578085c029224677fc463aa62c7c334b800280a2210daf457d"} Nov 02 19:49:03.808554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:03.808216 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" event={"ID":"67118bc7-f99c-47f8-894d-8d82423e1435","Type":"ContainerStarted","Data":"c52a875b5397dedf7893eaac9a6d3e8d0d09d423bc6cb2d1b644b6cf34c6bd75"} Nov 02 19:49:03.917674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:03.917623 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" podStartSLOduration=1.917606937 podStartE2EDuration="1.917606937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:03.91229094 +0000 UTC m=+1249.860946790" watchObservedRunningTime="2026-11-02 19:49:03.917606937 +0000 UTC m=+1249.866262777" Nov 02 19:49:04.045138 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:04.045104 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k"] Nov 02 19:49:04.049689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:04.049657 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e331cba-5l25k"] Nov 02 19:49:04.185974 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:04.185887 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292f2ff6-572c-404d-bcb0-af60496177c5" path="/var/lib/kubelet/pods/292f2ff6-572c-404d-bcb0-af60496177c5/volumes" Nov 02 19:49:04.818999 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:04.818953 4966 generic.go:356] "Generic (PLEG): container finished" podID="67118bc7-f99c-47f8-894d-8d82423e1435" containerID="b77206b6a3a0f5578085c029224677fc463aa62c7c334b800280a2210daf457d" exitCode=0 Nov 02 19:49:04.819505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:04.819007 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" event={"ID":"67118bc7-f99c-47f8-894d-8d82423e1435","Type":"ContainerDied","Data":"b77206b6a3a0f5578085c029224677fc463aa62c7c334b800280a2210daf457d"} Nov 02 19:49:05.046639 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.046600 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft"] Nov 02 19:49:05.051219 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.051193 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.108674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.108594 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft"] Nov 02 19:49:05.140061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140022 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140070 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhl6p\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-kube-api-access-bhl6p\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140341 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140183 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140341 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140240 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140341 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140281 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140386 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140481 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140627 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140606 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.140680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.140668 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbf3a24-63a8-4e99-898d-4a95efa38833-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.241826 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.241787 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.241850 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbf3a24-63a8-4e99-898d-4a95efa38833-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.241936 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.241962 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhl6p\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-kube-api-access-bhl6p\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242004 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242027 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242059 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242117 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242210 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242473 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242255 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242654 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242634 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbf3a24-63a8-4e99-898d-4a95efa38833-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.242854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242818 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.243010 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.242983 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.243137 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.243068 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.244712 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.244689 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.244980 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.244963 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.295418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.295387 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.295539 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.295455 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhl6p\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-kube-api-access-bhl6p\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.368698 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.368636 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:05.457258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.455767 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn"] Nov 02 19:49:05.462682 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.462653 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.515343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.509773 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn"] Nov 02 19:49:05.549885 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.549842 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.550051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.550026 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bdd\" (UniqueName: \"kubernetes.io/projected/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-kube-api-access-j9bdd\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.550191 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.550173 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.651539 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.651443 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.651709 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.651599 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.651788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.651737 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bdd\" (UniqueName: \"kubernetes.io/projected/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-kube-api-access-j9bdd\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.652653 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.652627 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.654972 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.654948 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.704595 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.704568 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bdd\" (UniqueName: \"kubernetes.io/projected/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-kube-api-access-j9bdd\") pod \"maas-api-revoke-sub-63b42fa7-zw8zn\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.789353 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.789312 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:05.789819 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:05.789786 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbf3a24_63a8_4e99_898d_4a95efa38833.slice/crio-0772c9939c440a5f6345b4e892eac61ab9371e0c34ece84af7c465d787a56e0a WatchSource:0}: Error finding container 0772c9939c440a5f6345b4e892eac61ab9371e0c34ece84af7c465d787a56e0a: Status 404 returned error can't find the container with id 0772c9939c440a5f6345b4e892eac61ab9371e0c34ece84af7c465d787a56e0a Nov 02 19:49:05.792365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.792338 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft"] Nov 02 19:49:05.792649 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.792608 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:05.792746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.792687 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:05.792746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.792728 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:05.830540 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:05.830512 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" event={"ID":"acbf3a24-63a8-4e99-898d-4a95efa38833","Type":"ContainerStarted","Data":"0772c9939c440a5f6345b4e892eac61ab9371e0c34ece84af7c465d787a56e0a"} Nov 02 19:49:06.072092 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.072063 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:06.161521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.161489 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67118bc7-f99c-47f8-894d-8d82423e1435-kube-api-access-fmh2x\" (UniqueName: \"kubernetes.io/projected/67118bc7-f99c-47f8-894d-8d82423e1435-kube-api-access-fmh2x\") pod \"67118bc7-f99c-47f8-894d-8d82423e1435\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " Nov 02 19:49:06.161700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.161541 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-openshift-ca\") pod \"67118bc7-f99c-47f8-894d-8d82423e1435\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " Nov 02 19:49:06.161984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.161944 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-openshift-ca" pod "67118bc7-f99c-47f8-894d-8d82423e1435" (UID: "67118bc7-f99c-47f8-894d-8d82423e1435"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:06.162114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.162073 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-xks-ca\") pod \"67118bc7-f99c-47f8-894d-8d82423e1435\" (UID: \"67118bc7-f99c-47f8-894d-8d82423e1435\") " Nov 02 19:49:06.162615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.162590 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:06.163917 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.163887 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67118bc7-f99c-47f8-894d-8d82423e1435-kube-api-access-fmh2x" pod "67118bc7-f99c-47f8-894d-8d82423e1435" (UID: "67118bc7-f99c-47f8-894d-8d82423e1435"). InnerVolumeSpecName "kube-api-access-fmh2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:06.164090 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.164071 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-xks-ca" pod "67118bc7-f99c-47f8-894d-8d82423e1435" (UID: "67118bc7-f99c-47f8-894d-8d82423e1435"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:06.189248 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:06.189012 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fd34c6_64ed_4083_a4ac_f8dfa279fbe1.slice/crio-1591ba42eddf7470602a836859fad799441ac7560d25212e8f31bf430dd2125d WatchSource:0}: Error finding container 1591ba42eddf7470602a836859fad799441ac7560d25212e8f31bf430dd2125d: Status 404 returned error can't find the container with id 1591ba42eddf7470602a836859fad799441ac7560d25212e8f31bf430dd2125d Nov 02 19:49:06.201833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.200328 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn"] Nov 02 19:49:06.265628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.265593 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmh2x\" (UniqueName: \"kubernetes.io/projected/67118bc7-f99c-47f8-894d-8d82423e1435-kube-api-access-fmh2x\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:06.265844 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.265828 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67118bc7-f99c-47f8-894d-8d82423e1435-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:06.867423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.862465 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" event={"ID":"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1","Type":"ContainerStarted","Data":"2c7fc69641c46d44563c29e724c3c878468db97ddcfe24e687a0cb6b642dc6b2"} Nov 02 19:49:06.867423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.862558 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" event={"ID":"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1","Type":"ContainerStarted","Data":"1591ba42eddf7470602a836859fad799441ac7560d25212e8f31bf430dd2125d"} Nov 02 19:49:06.874367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.870920 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" event={"ID":"67118bc7-f99c-47f8-894d-8d82423e1435","Type":"ContainerDied","Data":"c52a875b5397dedf7893eaac9a6d3e8d0d09d423bc6cb2d1b644b6cf34c6bd75"} Nov 02 19:49:06.874367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.870985 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c52a875b5397dedf7893eaac9a6d3e8d0d09d423bc6cb2d1b644b6cf34c6bd75" Nov 02 19:49:06.874367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.871070 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks" Nov 02 19:49:06.881347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:06.880445 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" event={"ID":"acbf3a24-63a8-4e99-898d-4a95efa38833","Type":"ContainerStarted","Data":"dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89"} Nov 02 19:49:07.019113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.019056 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" podStartSLOduration=2.019040486 podStartE2EDuration="2.019040486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:06.969973296 +0000 UTC m=+1252.918629135" watchObservedRunningTime="2026-11-02 19:49:07.019040486 +0000 UTC m=+1252.967696324" Nov 02 19:49:07.082473 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.082416 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podStartSLOduration=3.082399235 podStartE2EDuration="3.082399235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:07.07912163 +0000 UTC m=+1253.027777479" watchObservedRunningTime="2026-11-02 19:49:07.082399235 +0000 UTC m=+1253.031055073" Nov 02 19:49:07.371511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.371478 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:07.374519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.374486 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 19:49:07.374759 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.374722 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 19:49:07.476410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.476364 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj"] Nov 02 19:49:07.478028 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.477418 4966 cpu_manager.go:414] "containerMap: removing container" podUID="67118bc7-f99c-47f8-894d-8d82423e1435" containerName="revoke-keys" Nov 02 19:49:07.478028 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.477442 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67118bc7-f99c-47f8-894d-8d82423e1435" containerName="revoke-keys" Nov 02 19:49:07.478028 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.477620 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="67118bc7-f99c-47f8-894d-8d82423e1435" containerName="revoke-keys" Nov 02 19:49:07.482839 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.482810 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.517039 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.517011 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj"] Nov 02 19:49:07.592352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.591414 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jwm\" (UniqueName: \"kubernetes.io/projected/eefd1c16-7116-4f45-af0d-49b44fdba4f8-kube-api-access-77jwm\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.592352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.591543 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.592352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.591645 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.693105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.693026 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.693264 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.693152 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jwm\" (UniqueName: \"kubernetes.io/projected/eefd1c16-7116-4f45-af0d-49b44fdba4f8-kube-api-access-77jwm\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.693264 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.693228 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.693740 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.693718 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.695643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.695624 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.758320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.755924 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jwm\" (UniqueName: \"kubernetes.io/projected/eefd1c16-7116-4f45-af0d-49b44fdba4f8-kube-api-access-77jwm\") pod \"maas-api-revoke-sub-8dbe1dd6-vwzxj\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.800994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.800957 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:07.898967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.898132 4966 generic.go:356] "Generic (PLEG): container finished" podID="20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" containerID="2c7fc69641c46d44563c29e724c3c878468db97ddcfe24e687a0cb6b642dc6b2" exitCode=0 Nov 02 19:49:07.899211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:07.899152 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" event={"ID":"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1","Type":"ContainerDied","Data":"2c7fc69641c46d44563c29e724c3c878468db97ddcfe24e687a0cb6b642dc6b2"} Nov 02 19:49:08.214441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.214290 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj"] Nov 02 19:49:08.373231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.373191 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 19:49:08.373533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.373499 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 19:49:08.493320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.482261 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g"] Nov 02 19:49:08.493520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.493350 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.556331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.553984 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g"] Nov 02 19:49:08.586798 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.586746 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x"] Nov 02 19:49:08.606746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.605038 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.610750 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.609162 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.610750 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.609481 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28sv\" (UniqueName: \"kubernetes.io/projected/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-kube-api-access-x28sv\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.610750 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.609515 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.631858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.628289 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x"] Nov 02 19:49:08.712070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.710926 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.712070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.710981 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28sv\" (UniqueName: \"kubernetes.io/projected/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-kube-api-access-x28sv\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.712070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.711013 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.712070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.711139 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.712070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.711210 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27x6r\" (UniqueName: \"kubernetes.io/projected/54557b95-d415-4f48-a5e1-63cc56569d31-kube-api-access-27x6r\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.712070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.711355 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.712632 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.712384 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.714766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.714733 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.785935 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.785190 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28sv\" (UniqueName: \"kubernetes.io/projected/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-kube-api-access-x28sv\") pod \"maas-api-revoke-sub-dd7e4a37-zwc9g\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.812675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.812649 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27x6r\" (UniqueName: \"kubernetes.io/projected/54557b95-d415-4f48-a5e1-63cc56569d31-kube-api-access-27x6r\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.812892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.812878 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.813041 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.813030 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.813695 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.813676 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.814710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.814696 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:08.818184 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.817912 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.882335 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.881455 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27x6r\" (UniqueName: \"kubernetes.io/projected/54557b95-d415-4f48-a5e1-63cc56569d31-kube-api-access-27x6r\") pod \"maas-api-revoke-sub-9c2748ff-qlw5x\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:08.919737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.918498 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" event={"ID":"eefd1c16-7116-4f45-af0d-49b44fdba4f8","Type":"ContainerStarted","Data":"83f94295d7a9f1538954ebe0947ca4471b9ee5ad2be4e5d1308f312d07ad3627"} Nov 02 19:49:08.919737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.918613 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" event={"ID":"eefd1c16-7116-4f45-af0d-49b44fdba4f8","Type":"ContainerStarted","Data":"64d743daa54e6f7690d78c2ab69df6a5cb36b65fd8aa420be231339b8f766f7c"} Nov 02 19:49:08.928492 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:08.928400 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:09.077511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.037014 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" podStartSLOduration=2.036994709 podStartE2EDuration="2.036994709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:09.036420195 +0000 UTC m=+1254.985076034" watchObservedRunningTime="2026-11-02 19:49:09.036994709 +0000 UTC m=+1254.985650550" Nov 02 19:49:09.285363 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:09.264028 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa08fb5d_e3bd_45c4_af7d_a54fbd1499ae.slice/crio-27ad62e1e5036d9153ed263c22b49834a0120d01335e66e7c3aa153020d2372e WatchSource:0}: Error finding container 27ad62e1e5036d9153ed263c22b49834a0120d01335e66e7c3aa153020d2372e: Status 404 returned error can't find the container with id 27ad62e1e5036d9153ed263c22b49834a0120d01335e66e7c3aa153020d2372e Nov 02 19:49:09.296800 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.295813 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g"] Nov 02 19:49:09.325501 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.325442 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:09.392606 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.392334 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 19:49:09.392606 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.392389 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 19:49:09.425018 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.424979 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-kube-api-access-j9bdd\" (UniqueName: \"kubernetes.io/projected/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-kube-api-access-j9bdd\") pod \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " Nov 02 19:49:09.425180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.425156 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-xks-ca\") pod \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " Nov 02 19:49:09.425454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.425432 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-openshift-ca\") pod \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\" (UID: \"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1\") " Nov 02 19:49:09.426541 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.426514 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-openshift-ca" pod "20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" (UID: "20fd34c6-64ed-4083-a4ac-f8dfa279fbe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:09.499947 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.499889 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-kube-api-access-j9bdd" pod "20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" (UID: "20fd34c6-64ed-4083-a4ac-f8dfa279fbe1"). InnerVolumeSpecName "kube-api-access-j9bdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:09.508350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.508282 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-xks-ca" pod "20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" (UID: "20fd34c6-64ed-4083-a4ac-f8dfa279fbe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:09.527438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.527364 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:09.527438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.527400 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:09.527438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.527415 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9bdd\" (UniqueName: \"kubernetes.io/projected/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1-kube-api-access-j9bdd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:09.695044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.695009 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x"] Nov 02 19:49:09.928754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.928665 4966 generic.go:356] "Generic (PLEG): container finished" podID="eefd1c16-7116-4f45-af0d-49b44fdba4f8" containerID="83f94295d7a9f1538954ebe0947ca4471b9ee5ad2be4e5d1308f312d07ad3627" exitCode=0 Nov 02 19:49:09.929224 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.928786 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" event={"ID":"eefd1c16-7116-4f45-af0d-49b44fdba4f8","Type":"ContainerDied","Data":"83f94295d7a9f1538954ebe0947ca4471b9ee5ad2be4e5d1308f312d07ad3627"} Nov 02 19:49:09.930755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.930727 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" event={"ID":"54557b95-d415-4f48-a5e1-63cc56569d31","Type":"ContainerStarted","Data":"167bd268ec257998e76516c52dc6be02eb9e8cd15f9e635c4f97be0bb2502b19"} Nov 02 19:49:09.932158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.932134 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" event={"ID":"20fd34c6-64ed-4083-a4ac-f8dfa279fbe1","Type":"ContainerDied","Data":"1591ba42eddf7470602a836859fad799441ac7560d25212e8f31bf430dd2125d"} Nov 02 19:49:09.932276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.932178 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1591ba42eddf7470602a836859fad799441ac7560d25212e8f31bf430dd2125d" Nov 02 19:49:09.932276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.932243 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn" Nov 02 19:49:09.934291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.934263 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" event={"ID":"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae","Type":"ContainerStarted","Data":"c8cfb9ad2debeecdbde675a137b420827e04d46c8777748918373421c1a547b7"} Nov 02 19:49:09.934436 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:09.934324 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" event={"ID":"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae","Type":"ContainerStarted","Data":"27ad62e1e5036d9153ed263c22b49834a0120d01335e66e7c3aa153020d2372e"} Nov 02 19:49:10.123908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.123871 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr"] Nov 02 19:49:10.124904 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.124878 4966 cpu_manager.go:414] "containerMap: removing container" podUID="20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" containerName="revoke-keys" Nov 02 19:49:10.124904 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.124904 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" containerName="revoke-keys" Nov 02 19:49:10.125088 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.125076 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" containerName="revoke-keys" Nov 02 19:49:10.129440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.129420 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.195214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.191377 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr"] Nov 02 19:49:10.216130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.216070 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" podStartSLOduration=2.216050339 podStartE2EDuration="2.216050339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:10.212496838 +0000 UTC m=+1256.161152679" watchObservedRunningTime="2026-11-02 19:49:10.216050339 +0000 UTC m=+1256.164706180" Nov 02 19:49:10.300359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.299689 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.300359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.299797 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.300359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.300352 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtsf2\" (UniqueName: \"kubernetes.io/projected/ce7697a9-283a-44f0-9f4a-7e456dcea93e-kube-api-access-qtsf2\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.378803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.378653 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 19:49:10.378803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.378712 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 19:49:10.402660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.401581 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtsf2\" (UniqueName: \"kubernetes.io/projected/ce7697a9-283a-44f0-9f4a-7e456dcea93e-kube-api-access-qtsf2\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.402660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.401764 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.402660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.401793 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.402660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.402443 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.404671 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.404652 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.476433 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.476400 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtsf2\" (UniqueName: \"kubernetes.io/projected/ce7697a9-283a-44f0-9f4a-7e456dcea93e-kube-api-access-qtsf2\") pod \"maas-api-revoke-sub-3f6742d7-4jjzr\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:10.782820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:10.782715 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:11.005947 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.005904 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" event={"ID":"54557b95-d415-4f48-a5e1-63cc56569d31","Type":"ContainerStarted","Data":"1bcadf5972446e8e9ebb94105d558677d52ba58cc16f36b3b5adec4f34046b71"} Nov 02 19:49:11.016398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.016340 4966 generic.go:356] "Generic (PLEG): container finished" podID="aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" containerID="c8cfb9ad2debeecdbde675a137b420827e04d46c8777748918373421c1a547b7" exitCode=0 Nov 02 19:49:11.016547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.016490 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" event={"ID":"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae","Type":"ContainerDied","Data":"c8cfb9ad2debeecdbde675a137b420827e04d46c8777748918373421c1a547b7"} Nov 02 19:49:11.190983 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.190922 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" podStartSLOduration=3.190901851 podStartE2EDuration="3.190901851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:11.182612952 +0000 UTC m=+1257.131268792" watchObservedRunningTime="2026-11-02 19:49:11.190901851 +0000 UTC m=+1257.139557691" Nov 02 19:49:11.288474 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.288441 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr"] Nov 02 19:49:11.289146 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:11.288669 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7697a9_283a_44f0_9f4a_7e456dcea93e.slice/crio-6d8bc0eff5f12f22e4e5259580f7fa342eda5d37e14820572672a9a294ddab7c WatchSource:0}: Error finding container 6d8bc0eff5f12f22e4e5259580f7fa342eda5d37e14820572672a9a294ddab7c: Status 404 returned error can't find the container with id 6d8bc0eff5f12f22e4e5259580f7fa342eda5d37e14820572672a9a294ddab7c Nov 02 19:49:11.314470 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.314441 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:11.370343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.369977 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 19:49:11.370343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.370041 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 19:49:11.421270 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.421243 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-xks-ca\") pod \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " Nov 02 19:49:11.421422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.421336 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-openshift-ca\") pod \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " Nov 02 19:49:11.421507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.421484 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefd1c16-7116-4f45-af0d-49b44fdba4f8-kube-api-access-77jwm\" (UniqueName: \"kubernetes.io/projected/eefd1c16-7116-4f45-af0d-49b44fdba4f8-kube-api-access-77jwm\") pod \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\" (UID: \"eefd1c16-7116-4f45-af0d-49b44fdba4f8\") " Nov 02 19:49:11.421827 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.421791 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-openshift-ca" pod "eefd1c16-7116-4f45-af0d-49b44fdba4f8" (UID: "eefd1c16-7116-4f45-af0d-49b44fdba4f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:11.422214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.422195 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:11.423657 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.423634 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefd1c16-7116-4f45-af0d-49b44fdba4f8-kube-api-access-77jwm" pod "eefd1c16-7116-4f45-af0d-49b44fdba4f8" (UID: "eefd1c16-7116-4f45-af0d-49b44fdba4f8"). InnerVolumeSpecName "kube-api-access-77jwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:11.423986 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.423965 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-xks-ca" pod "eefd1c16-7116-4f45-af0d-49b44fdba4f8" (UID: "eefd1c16-7116-4f45-af0d-49b44fdba4f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:11.523622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.523584 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefd1c16-7116-4f45-af0d-49b44fdba4f8-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:11.523869 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:11.523839 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jwm\" (UniqueName: \"kubernetes.io/projected/eefd1c16-7116-4f45-af0d-49b44fdba4f8-kube-api-access-77jwm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:12.026282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.026247 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" event={"ID":"ce7697a9-283a-44f0-9f4a-7e456dcea93e","Type":"ContainerStarted","Data":"e9f75a606924babdac73154df40b7dc96bd8775da1b406178273766df2500ed9"} Nov 02 19:49:12.026697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.026320 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" event={"ID":"ce7697a9-283a-44f0-9f4a-7e456dcea93e","Type":"ContainerStarted","Data":"6d8bc0eff5f12f22e4e5259580f7fa342eda5d37e14820572672a9a294ddab7c"} Nov 02 19:49:12.027852 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.027831 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" Nov 02 19:49:12.027984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.027861 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj" event={"ID":"eefd1c16-7116-4f45-af0d-49b44fdba4f8","Type":"ContainerDied","Data":"64d743daa54e6f7690d78c2ab69df6a5cb36b65fd8aa420be231339b8f766f7c"} Nov 02 19:49:12.027984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.027902 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64d743daa54e6f7690d78c2ab69df6a5cb36b65fd8aa420be231339b8f766f7c" Nov 02 19:49:12.029783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.029758 4966 generic.go:356] "Generic (PLEG): container finished" podID="54557b95-d415-4f48-a5e1-63cc56569d31" containerID="1bcadf5972446e8e9ebb94105d558677d52ba58cc16f36b3b5adec4f34046b71" exitCode=0 Nov 02 19:49:12.029884 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.029791 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" event={"ID":"54557b95-d415-4f48-a5e1-63cc56569d31","Type":"ContainerDied","Data":"1bcadf5972446e8e9ebb94105d558677d52ba58cc16f36b3b5adec4f34046b71"} Nov 02 19:49:12.134800 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.131962 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" podStartSLOduration=2.131942392 podStartE2EDuration="2.131942392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:12.131586724 +0000 UTC m=+1258.080242563" watchObservedRunningTime="2026-11-02 19:49:12.131942392 +0000 UTC m=+1258.080598230" Nov 02 19:49:12.248415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.248381 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:12.333415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.333329 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-kube-api-access-x28sv\" (UniqueName: \"kubernetes.io/projected/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-kube-api-access-x28sv\") pod \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " Nov 02 19:49:12.333572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.333451 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-openshift-ca\") pod \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " Nov 02 19:49:12.333572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.333502 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-xks-ca\") pod \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\" (UID: \"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae\") " Nov 02 19:49:12.333791 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.333770 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-openshift-ca" pod "aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" (UID: "aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:12.334008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.333983 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:12.335662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.335638 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-xks-ca" pod "aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" (UID: "aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:12.335754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.335720 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-kube-api-access-x28sv" pod "aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" (UID: "aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae"). InnerVolumeSpecName "kube-api-access-x28sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:12.369222 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.369195 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 19:49:12.369346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.369240 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 19:49:12.435184 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.435153 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:12.435184 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:12.435183 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28sv\" (UniqueName: \"kubernetes.io/projected/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae-kube-api-access-x28sv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:13.044361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.043518 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" Nov 02 19:49:13.044361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.043525 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g" event={"ID":"aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae","Type":"ContainerDied","Data":"27ad62e1e5036d9153ed263c22b49834a0120d01335e66e7c3aa153020d2372e"} Nov 02 19:49:13.044361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.043676 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ad62e1e5036d9153ed263c22b49834a0120d01335e66e7c3aa153020d2372e" Nov 02 19:49:13.046046 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.046016 4966 generic.go:356] "Generic (PLEG): container finished" podID="ce7697a9-283a-44f0-9f4a-7e456dcea93e" containerID="e9f75a606924babdac73154df40b7dc96bd8775da1b406178273766df2500ed9" exitCode=0 Nov 02 19:49:13.046660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.046636 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" event={"ID":"ce7697a9-283a-44f0-9f4a-7e456dcea93e","Type":"ContainerDied","Data":"e9f75a606924babdac73154df40b7dc96bd8775da1b406178273766df2500ed9"} Nov 02 19:49:13.274570 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.274541 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:13.348130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.348045 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-xks-ca\") pod \"54557b95-d415-4f48-a5e1-63cc56569d31\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " Nov 02 19:49:13.348317 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.348285 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-openshift-ca\") pod \"54557b95-d415-4f48-a5e1-63cc56569d31\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " Nov 02 19:49:13.348403 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.348364 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54557b95-d415-4f48-a5e1-63cc56569d31-kube-api-access-27x6r\" (UniqueName: \"kubernetes.io/projected/54557b95-d415-4f48-a5e1-63cc56569d31-kube-api-access-27x6r\") pod \"54557b95-d415-4f48-a5e1-63cc56569d31\" (UID: \"54557b95-d415-4f48-a5e1-63cc56569d31\") " Nov 02 19:49:13.348704 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.348670 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-openshift-ca" pod "54557b95-d415-4f48-a5e1-63cc56569d31" (UID: "54557b95-d415-4f48-a5e1-63cc56569d31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:13.348979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.348956 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:13.350541 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.350514 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54557b95-d415-4f48-a5e1-63cc56569d31-kube-api-access-27x6r" pod "54557b95-d415-4f48-a5e1-63cc56569d31" (UID: "54557b95-d415-4f48-a5e1-63cc56569d31"). InnerVolumeSpecName "kube-api-access-27x6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:13.350721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.350698 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-xks-ca" pod "54557b95-d415-4f48-a5e1-63cc56569d31" (UID: "54557b95-d415-4f48-a5e1-63cc56569d31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:13.450198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.450154 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27x6r\" (UniqueName: \"kubernetes.io/projected/54557b95-d415-4f48-a5e1-63cc56569d31-kube-api-access-27x6r\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:13.450198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.450190 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54557b95-d415-4f48-a5e1-63cc56569d31-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:13.543278 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.543241 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:13.543820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.543800 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:13.544591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:13.544567 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 19:49:14.085322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.079126 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" Nov 02 19:49:14.088404 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.088375 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x" event={"ID":"54557b95-d415-4f48-a5e1-63cc56569d31","Type":"ContainerDied","Data":"167bd268ec257998e76516c52dc6be02eb9e8cd15f9e635c4f97be0bb2502b19"} Nov 02 19:49:14.088586 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.088570 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="167bd268ec257998e76516c52dc6be02eb9e8cd15f9e635c4f97be0bb2502b19" Nov 02 19:49:14.393516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.393462 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:14.466138 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.466094 4966 scope.go:122] "RemoveContainer" containerID="ac3ade01ad55587bdc9cd733dab008814b5a2ca4f8e5e37f540b04ebb71a8630" Nov 02 19:49:14.493469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.493443 4966 scope.go:122] "RemoveContainer" containerID="e3ecfc0e459d4501b0ff318bbc0a34f1df64dfa1fc4926cbd85330499519a9ad" Nov 02 19:49:14.617879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.617779 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7697a9-283a-44f0-9f4a-7e456dcea93e-kube-api-access-qtsf2\" (UniqueName: \"kubernetes.io/projected/ce7697a9-283a-44f0-9f4a-7e456dcea93e-kube-api-access-qtsf2\") pod \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " Nov 02 19:49:14.652324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.634280 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-xks-ca\") pod \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " Nov 02 19:49:14.652901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.652868 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-openshift-ca\") pod \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\" (UID: \"ce7697a9-283a-44f0-9f4a-7e456dcea93e\") " Nov 02 19:49:14.654030 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.633766 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7697a9-283a-44f0-9f4a-7e456dcea93e-kube-api-access-qtsf2" pod "ce7697a9-283a-44f0-9f4a-7e456dcea93e" (UID: "ce7697a9-283a-44f0-9f4a-7e456dcea93e"). InnerVolumeSpecName "kube-api-access-qtsf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:14.656422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.656392 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-openshift-ca" pod "ce7697a9-283a-44f0-9f4a-7e456dcea93e" (UID: "ce7697a9-283a-44f0-9f4a-7e456dcea93e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:14.689617 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.689513 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-xks-ca" pod "ce7697a9-283a-44f0-9f4a-7e456dcea93e" (UID: "ce7697a9-283a-44f0-9f4a-7e456dcea93e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:14.728315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.724618 4966 scope.go:122] "RemoveContainer" containerID="09c958240457da38b609535b470935d5b18003c3281777025bf1b354a7670c13" Nov 02 19:49:14.803969 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.799063 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:14.803969 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.799111 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtsf2\" (UniqueName: \"kubernetes.io/projected/ce7697a9-283a-44f0-9f4a-7e456dcea93e-kube-api-access-qtsf2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:14.803969 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:14.799127 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7697a9-283a-44f0-9f4a-7e456dcea93e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:15.100357 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:15.033492 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:49:15.261323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.254848 4966 scope.go:122] "RemoveContainer" containerID="bc79a2ee7dc7764d63129defd514ffe3e87f7d73bbfbd1b400649d3b8fd5399e" Nov 02 19:49:15.337329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.336873 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" event={"ID":"ce7697a9-283a-44f0-9f4a-7e456dcea93e","Type":"ContainerDied","Data":"6d8bc0eff5f12f22e4e5259580f7fa342eda5d37e14820572672a9a294ddab7c"} Nov 02 19:49:15.337329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.336931 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8bc0eff5f12f22e4e5259580f7fa342eda5d37e14820572672a9a294ddab7c" Nov 02 19:49:15.337329 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.336993 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr" Nov 02 19:49:15.446506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.446444 4966 scope.go:122] "RemoveContainer" containerID="fd50aa0c0a7522654a86f48f3fbc9af9ee9d966c5f84406a9bf8bd6bd5be73a8" Nov 02 19:49:15.602855 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.602825 4966 scope.go:122] "RemoveContainer" containerID="57a7ae8712faa8e0c6d1ac5b962d21d932845af5f52e76e32c4ff31c2cd53c94" Nov 02 19:49:15.796319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.795485 4966 scope.go:122] "RemoveContainer" containerID="e777da9dbace88475f623b198aedf4fb673daa8b72fc7a48d154944bdbcdc607" Nov 02 19:49:15.933622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.924381 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:15.967666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:15.967625 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.245327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.240751 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 02 19:49:16.245327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.241064 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:49:16.292718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.289823 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 19:49:16.315947 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.315434 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.456410 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.456265 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.472587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.472546 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.473111 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.472805 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.612324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.611443 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.612324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.611831 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 19:49:16.640325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.639379 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.946486 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.946429 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 19:49:16.947463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.947426 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:49:16.968938 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.968888 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:16.969555 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:16.969533 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:49:17.259447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:17.257115 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:49:17.261953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:17.261027 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:49:19.228766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.228727 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x"] Nov 02 19:49:19.229603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229581 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ce7697a9-283a-44f0-9f4a-7e456dcea93e" containerName="revoke-keys" Nov 02 19:49:19.229603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229603 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7697a9-283a-44f0-9f4a-7e456dcea93e" containerName="revoke-keys" Nov 02 19:49:19.229787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229638 4966 cpu_manager.go:414] "containerMap: removing container" podUID="54557b95-d415-4f48-a5e1-63cc56569d31" containerName="revoke-keys" Nov 02 19:49:19.229787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229646 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54557b95-d415-4f48-a5e1-63cc56569d31" containerName="revoke-keys" Nov 02 19:49:19.229787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229663 4966 cpu_manager.go:414] "containerMap: removing container" podUID="aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" containerName="revoke-keys" Nov 02 19:49:19.229787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229671 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" containerName="revoke-keys" Nov 02 19:49:19.229787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229710 4966 cpu_manager.go:414] "containerMap: removing container" podUID="eefd1c16-7116-4f45-af0d-49b44fdba4f8" containerName="revoke-keys" Nov 02 19:49:19.229787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229718 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefd1c16-7116-4f45-af0d-49b44fdba4f8" containerName="revoke-keys" Nov 02 19:49:19.230067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229885 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7697a9-283a-44f0-9f4a-7e456dcea93e" containerName="revoke-keys" Nov 02 19:49:19.230067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229906 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="54557b95-d415-4f48-a5e1-63cc56569d31" containerName="revoke-keys" Nov 02 19:49:19.230067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229917 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" containerName="revoke-keys" Nov 02 19:49:19.230067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.229929 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefd1c16-7116-4f45-af0d-49b44fdba4f8" containerName="revoke-keys" Nov 02 19:49:19.235773 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.235748 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.287726 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.287699 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x"] Nov 02 19:49:19.326580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.326549 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:49:19.326743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.326595 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:49:19.326743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.326607 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:49:19.326743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.326598 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:49:19.329399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.329368 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.329600 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.329445 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgjb\" (UniqueName: \"kubernetes.io/projected/8f20784f-9892-49f3-b784-c2837fa2afdf-kube-api-access-gkgjb\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.329600 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.329541 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.335805 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.335782 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2"] Nov 02 19:49:19.340766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.340746 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.394779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.394743 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2"] Nov 02 19:49:19.430921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.430887 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgjb\" (UniqueName: \"kubernetes.io/projected/8f20784f-9892-49f3-b784-c2837fa2afdf-kube-api-access-gkgjb\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.431109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.430941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.431109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.430999 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.431109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.431087 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92g6\" (UniqueName: \"kubernetes.io/projected/c970045e-af95-4c6b-be41-065f4837caaa-kube-api-access-w92g6\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.431291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.431245 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.431452 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.431429 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.432267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.432245 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.433507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.433459 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.484034 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.483960 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgjb\" (UniqueName: \"kubernetes.io/projected/8f20784f-9892-49f3-b784-c2837fa2afdf-kube-api-access-gkgjb\") pod \"maas-api-revoke-sub-83cd1fcc-btr9x\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.532782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.532749 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.532962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.532820 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92g6\" (UniqueName: \"kubernetes.io/projected/c970045e-af95-4c6b-be41-065f4837caaa-kube-api-access-w92g6\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.532962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.532883 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.533486 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.533460 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.535179 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.535158 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.550353 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.550320 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:19.586006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.585979 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92g6\" (UniqueName: \"kubernetes.io/projected/c970045e-af95-4c6b-be41-065f4837caaa-kube-api-access-w92g6\") pod \"maas-api-revoke-sub-434d6db5-94gm2\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.612356 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.612313 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7137-30af7c"] Nov 02 19:49:19.618983 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.618961 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:19.657832 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.656679 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:19.675881 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.672225 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-30af7c"] Nov 02 19:49:19.675881 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.672547 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cwvtz\"" Nov 02 19:49:19.735732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.735674 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz94v\" (UniqueName: \"kubernetes.io/projected/0053b615-e806-46bd-9522-a6e7f3c39c8d-kube-api-access-bz94v\") pod \"test-curl-7137-30af7c\" (UID: \"0053b615-e806-46bd-9522-a6e7f3c39c8d\") " pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:19.838630 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.838590 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz94v\" (UniqueName: \"kubernetes.io/projected/0053b615-e806-46bd-9522-a6e7f3c39c8d-kube-api-access-bz94v\") pod \"test-curl-7137-30af7c\" (UID: \"0053b615-e806-46bd-9522-a6e7f3c39c8d\") " pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:19.892093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.892049 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz94v\" (UniqueName: \"kubernetes.io/projected/0053b615-e806-46bd-9522-a6e7f3c39c8d-kube-api-access-bz94v\") pod \"test-curl-7137-30af7c\" (UID: \"0053b615-e806-46bd-9522-a6e7f3c39c8d\") " pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:19.935576 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:19.933768 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f20784f_9892_49f3_b784_c2837fa2afdf.slice/crio-7bb711986e2a03e79a6a61bfbd278c924d9dfe5bf13899927f4afdd545fe5273 WatchSource:0}: Error finding container 7bb711986e2a03e79a6a61bfbd278c924d9dfe5bf13899927f4afdd545fe5273: Status 404 returned error can't find the container with id 7bb711986e2a03e79a6a61bfbd278c924d9dfe5bf13899927f4afdd545fe5273 Nov 02 19:49:19.935923 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.935891 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x"] Nov 02 19:49:19.944439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.944412 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:19.959104 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.958808 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs"] Nov 02 19:49:19.965484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:19.965461 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.030126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.030097 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs"] Nov 02 19:49:20.058067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.057368 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2"] Nov 02 19:49:20.061725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.061698 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g"] Nov 02 19:49:20.062894 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:20.062478 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc970045e_af95_4c6b_be41_065f4837caaa.slice/crio-85465aac3efa595071f1324e490be55751f375ba63aaba96f4d526c9e9b5c114 WatchSource:0}: Error finding container 85465aac3efa595071f1324e490be55751f375ba63aaba96f4d526c9e9b5c114: Status 404 returned error can't find the container with id 85465aac3efa595071f1324e490be55751f375ba63aaba96f4d526c9e9b5c114 Nov 02 19:49:20.073629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.068713 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bc21bd-zzm6g"] Nov 02 19:49:20.142967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.142929 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143107 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143008 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143107 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143060 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267809e1-83dd-445f-8646-2a77e3b9bb97-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143177 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143135 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143266 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143405 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143386 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143457 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143443 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqrn\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-kube-api-access-rvqrn\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143543 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143482 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.143708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.143688 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.188095 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.188062 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5255ae7e-ac4e-44f8-904f-9e5796b65f33" path="/var/lib/kubelet/pods/5255ae7e-ac4e-44f8-904f-9e5796b65f33/volumes" Nov 02 19:49:20.245766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.245664 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267809e1-83dd-445f-8646-2a77e3b9bb97-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.245766 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.245771 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.245893 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.245999 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.246049 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqrn\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-kube-api-access-rvqrn\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.246085 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.246109 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.246170 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.246226 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.246747 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.246559 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267809e1-83dd-445f-8646-2a77e3b9bb97-istiod-ca-cert\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.247052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.246996 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-certs\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.247315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.247273 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.247924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.247902 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-credential-socket\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.248126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.248098 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-data\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.250211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.250186 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-podinfo\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.250529 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.250507 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-envoy\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.303496 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.303415 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-token\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.303705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.303627 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqrn\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-kube-api-access-rvqrn\") pod \"e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.333237 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:20.333206 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0053b615_e806_46bd_9522_a6e7f3c39c8d.slice/crio-d6db1fb977794be472016a9d334c541ba5c3d8525eedafa2f6d0c708f014cd00 WatchSource:0}: Error finding container d6db1fb977794be472016a9d334c541ba5c3d8525eedafa2f6d0c708f014cd00: Status 404 returned error can't find the container with id d6db1fb977794be472016a9d334c541ba5c3d8525eedafa2f6d0c708f014cd00 Nov 02 19:49:20.333876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.333852 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-30af7c"] Nov 02 19:49:20.335113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.335095 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:49:20.594272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.593506 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:20.621280 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.621251 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-30af7c" event={"ID":"0053b615-e806-46bd-9522-a6e7f3c39c8d","Type":"ContainerStarted","Data":"d6db1fb977794be472016a9d334c541ba5c3d8525eedafa2f6d0c708f014cd00"} Nov 02 19:49:20.623525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.623496 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" event={"ID":"8f20784f-9892-49f3-b784-c2837fa2afdf","Type":"ContainerStarted","Data":"df6ed0bb07c45587a056026ed352427846bed541c4a1fda335f393a9b2d7f0fd"} Nov 02 19:49:20.623646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.623554 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" event={"ID":"8f20784f-9892-49f3-b784-c2837fa2afdf","Type":"ContainerStarted","Data":"7bb711986e2a03e79a6a61bfbd278c924d9dfe5bf13899927f4afdd545fe5273"} Nov 02 19:49:20.625778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.625745 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" event={"ID":"c970045e-af95-4c6b-be41-065f4837caaa","Type":"ContainerStarted","Data":"5fe993b0ab84a6c428deec7481b5fe277c565bda0461f9f933af25038f282356"} Nov 02 19:49:20.625902 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.625790 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" event={"ID":"c970045e-af95-4c6b-be41-065f4837caaa","Type":"ContainerStarted","Data":"85465aac3efa595071f1324e490be55751f375ba63aaba96f4d526c9e9b5c114"} Nov 02 19:49:20.733343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.733275 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" podStartSLOduration=1.7332600550000001 podStartE2EDuration="1.733260055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:20.729773846 +0000 UTC m=+1266.678429699" watchObservedRunningTime="2026-11-02 19:49:20.733260055 +0000 UTC m=+1266.681915894" Nov 02 19:49:20.821666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.821633 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm"] Nov 02 19:49:20.827244 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.827224 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:20.843631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.842100 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" podStartSLOduration=1.8420873599999998 podStartE2EDuration="1.84208736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:20.836504405 +0000 UTC m=+1266.785160243" watchObservedRunningTime="2026-11-02 19:49:20.84208736 +0000 UTC m=+1266.790743199" Nov 02 19:49:20.892263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.892178 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm"] Nov 02 19:49:20.957214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.957170 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:20.957391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.957231 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwl4b\" (UniqueName: \"kubernetes.io/projected/37bb2374-6d61-4104-8112-2196ccb80fe7-kube-api-access-bwl4b\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:20.957581 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:20.957540 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.034286 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:21.034250 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267809e1_83dd_445f_8646_2a77e3b9bb97.slice/crio-06d5e192c92722a8afd2f000d38741c0abed588c0adcdbb30c418849cb0b8b4b WatchSource:0}: Error finding container 06d5e192c92722a8afd2f000d38741c0abed588c0adcdbb30c418849cb0b8b4b: Status 404 returned error can't find the container with id 06d5e192c92722a8afd2f000d38741c0abed588c0adcdbb30c418849cb0b8b4b Nov 02 19:49:21.037205 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.037169 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:21.037337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.037238 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:21.037337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.037269 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:21.037901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.037843 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs"] Nov 02 19:49:21.059764 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.059729 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.059993 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.059970 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.060093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.060013 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwl4b\" (UniqueName: \"kubernetes.io/projected/37bb2374-6d61-4104-8112-2196ccb80fe7-kube-api-access-bwl4b\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.061887 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.061862 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.063155 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.063128 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.113835 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.113792 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwl4b\" (UniqueName: \"kubernetes.io/projected/37bb2374-6d61-4104-8112-2196ccb80fe7-kube-api-access-bwl4b\") pod \"maas-api-revoke-sub-e8437d0d-jtrtm\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.147313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.147224 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:21.529018 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:21.528982 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bb2374_6d61_4104_8112_2196ccb80fe7.slice/crio-5eeb98cd4af36c38d9b50188dfc32abd656ab8886cfcac2ea9c4db7d6429348a WatchSource:0}: Error finding container 5eeb98cd4af36c38d9b50188dfc32abd656ab8886cfcac2ea9c4db7d6429348a: Status 404 returned error can't find the container with id 5eeb98cd4af36c38d9b50188dfc32abd656ab8886cfcac2ea9c4db7d6429348a Nov 02 19:49:21.529488 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.529217 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm"] Nov 02 19:49:21.653235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.653202 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" event={"ID":"267809e1-83dd-445f-8646-2a77e3b9bb97","Type":"ContainerStarted","Data":"832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a"} Nov 02 19:49:21.653551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.653525 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" event={"ID":"267809e1-83dd-445f-8646-2a77e3b9bb97","Type":"ContainerStarted","Data":"06d5e192c92722a8afd2f000d38741c0abed588c0adcdbb30c418849cb0b8b4b"} Nov 02 19:49:21.655440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.655415 4966 generic.go:356] "Generic (PLEG): container finished" podID="8f20784f-9892-49f3-b784-c2837fa2afdf" containerID="df6ed0bb07c45587a056026ed352427846bed541c4a1fda335f393a9b2d7f0fd" exitCode=0 Nov 02 19:49:21.655559 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.655501 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" event={"ID":"8f20784f-9892-49f3-b784-c2837fa2afdf","Type":"ContainerDied","Data":"df6ed0bb07c45587a056026ed352427846bed541c4a1fda335f393a9b2d7f0fd"} Nov 02 19:49:21.656951 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.656928 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" event={"ID":"37bb2374-6d61-4104-8112-2196ccb80fe7","Type":"ContainerStarted","Data":"5eeb98cd4af36c38d9b50188dfc32abd656ab8886cfcac2ea9c4db7d6429348a"} Nov 02 19:49:21.658745 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.658721 4966 generic.go:356] "Generic (PLEG): container finished" podID="c970045e-af95-4c6b-be41-065f4837caaa" containerID="5fe993b0ab84a6c428deec7481b5fe277c565bda0461f9f933af25038f282356" exitCode=0 Nov 02 19:49:21.658924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.658813 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" event={"ID":"c970045e-af95-4c6b-be41-065f4837caaa","Type":"ContainerDied","Data":"5fe993b0ab84a6c428deec7481b5fe277c565bda0461f9f933af25038f282356"} Nov 02 19:49:21.815323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:21.811099 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podStartSLOduration=2.811075191 podStartE2EDuration="2.811075191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:21.764837266 +0000 UTC m=+1267.713493109" watchObservedRunningTime="2026-11-02 19:49:21.811075191 +0000 UTC m=+1267.759731031" Nov 02 19:49:22.595503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:22.595468 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:22.596899 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:22.596869 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 19:49:22.597035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:22.596927 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 19:49:22.687496 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:22.687437 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" event={"ID":"37bb2374-6d61-4104-8112-2196ccb80fe7","Type":"ContainerStarted","Data":"e6f025d44b24493e374eb0ca33f25c74920953b2a9897ae47e67286e220a6d5c"} Nov 02 19:49:22.690548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:22.690512 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-30af7c" event={"ID":"0053b615-e806-46bd-9522-a6e7f3c39c8d","Type":"ContainerStarted","Data":"50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392"} Nov 02 19:49:22.924505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:22.924444 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7137-30af7c" podStartSLOduration=2.628959667 podStartE2EDuration="3.92442465s" totalImagesPullingTime="1.295464983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:22.917063107 +0000 UTC m=+1268.865718947" watchObservedRunningTime="2026-11-02 19:49:22.92442465 +0000 UTC m=+1268.873080492" Nov 02 19:49:22.982393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:22.980350 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:23.010019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.009518 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f20784f-9892-49f3-b784-c2837fa2afdf-kube-api-access-gkgjb\" (UniqueName: \"kubernetes.io/projected/8f20784f-9892-49f3-b784-c2837fa2afdf-kube-api-access-gkgjb\") pod \"8f20784f-9892-49f3-b784-c2837fa2afdf\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " Nov 02 19:49:23.010019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.009594 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-xks-ca\") pod \"8f20784f-9892-49f3-b784-c2837fa2afdf\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " Nov 02 19:49:23.010019 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.009890 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-openshift-ca\") pod \"8f20784f-9892-49f3-b784-c2837fa2afdf\" (UID: \"8f20784f-9892-49f3-b784-c2837fa2afdf\") " Nov 02 19:49:23.010742 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.010719 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-openshift-ca" pod "8f20784f-9892-49f3-b784-c2837fa2afdf" (UID: "8f20784f-9892-49f3-b784-c2837fa2afdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:23.011754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.011705 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f20784f-9892-49f3-b784-c2837fa2afdf-kube-api-access-gkgjb" pod "8f20784f-9892-49f3-b784-c2837fa2afdf" (UID: "8f20784f-9892-49f3-b784-c2837fa2afdf"). InnerVolumeSpecName "kube-api-access-gkgjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:23.012258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.012229 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-xks-ca" pod "8f20784f-9892-49f3-b784-c2837fa2afdf" (UID: "8f20784f-9892-49f3-b784-c2837fa2afdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:23.013973 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.013949 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:23.111508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.111461 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c970045e-af95-4c6b-be41-065f4837caaa-kube-api-access-w92g6\" (UniqueName: \"kubernetes.io/projected/c970045e-af95-4c6b-be41-065f4837caaa-kube-api-access-w92g6\") pod \"c970045e-af95-4c6b-be41-065f4837caaa\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " Nov 02 19:49:23.111721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.111620 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-xks-ca\") pod \"c970045e-af95-4c6b-be41-065f4837caaa\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " Nov 02 19:49:23.111721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.111686 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-openshift-ca\") pod \"c970045e-af95-4c6b-be41-065f4837caaa\" (UID: \"c970045e-af95-4c6b-be41-065f4837caaa\") " Nov 02 19:49:23.113841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.113810 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgjb\" (UniqueName: \"kubernetes.io/projected/8f20784f-9892-49f3-b784-c2837fa2afdf-kube-api-access-gkgjb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:23.114013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.113993 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:23.114136 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.114122 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f20784f-9892-49f3-b784-c2837fa2afdf-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:23.114686 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.114659 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-openshift-ca" pod "c970045e-af95-4c6b-be41-065f4837caaa" (UID: "c970045e-af95-4c6b-be41-065f4837caaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:23.116236 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.116181 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c970045e-af95-4c6b-be41-065f4837caaa-kube-api-access-w92g6" pod "c970045e-af95-4c6b-be41-065f4837caaa" (UID: "c970045e-af95-4c6b-be41-065f4837caaa"). InnerVolumeSpecName "kube-api-access-w92g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:23.117332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.117276 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-xks-ca" pod "c970045e-af95-4c6b-be41-065f4837caaa" (UID: "c970045e-af95-4c6b-be41-065f4837caaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:23.217372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.215072 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w92g6\" (UniqueName: \"kubernetes.io/projected/c970045e-af95-4c6b-be41-065f4837caaa-kube-api-access-w92g6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:23.217372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.215105 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:23.217372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.215115 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c970045e-af95-4c6b-be41-065f4837caaa-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:23.595953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.595525 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 19:49:23.595953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.595586 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 19:49:23.711810 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.711781 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" Nov 02 19:49:23.712215 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.712182 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x" event={"ID":"8f20784f-9892-49f3-b784-c2837fa2afdf","Type":"ContainerDied","Data":"7bb711986e2a03e79a6a61bfbd278c924d9dfe5bf13899927f4afdd545fe5273"} Nov 02 19:49:23.712441 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.712424 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bb711986e2a03e79a6a61bfbd278c924d9dfe5bf13899927f4afdd545fe5273" Nov 02 19:49:23.717572 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.717543 4966 generic.go:356] "Generic (PLEG): container finished" podID="37bb2374-6d61-4104-8112-2196ccb80fe7" containerID="e6f025d44b24493e374eb0ca33f25c74920953b2a9897ae47e67286e220a6d5c" exitCode=0 Nov 02 19:49:23.717688 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.717641 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" event={"ID":"37bb2374-6d61-4104-8112-2196ccb80fe7","Type":"ContainerDied","Data":"e6f025d44b24493e374eb0ca33f25c74920953b2a9897ae47e67286e220a6d5c"} Nov 02 19:49:23.722330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.722101 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" Nov 02 19:49:23.727586 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.723280 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2" event={"ID":"c970045e-af95-4c6b-be41-065f4837caaa","Type":"ContainerDied","Data":"85465aac3efa595071f1324e490be55751f375ba63aaba96f4d526c9e9b5c114"} Nov 02 19:49:23.727586 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:23.723346 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85465aac3efa595071f1324e490be55751f375ba63aaba96f4d526c9e9b5c114" Nov 02 19:49:24.128454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.128429 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:24.322318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.321966 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bb2374-6d61-4104-8112-2196ccb80fe7-kube-api-access-bwl4b\" (UniqueName: \"kubernetes.io/projected/37bb2374-6d61-4104-8112-2196ccb80fe7-kube-api-access-bwl4b\") pod \"37bb2374-6d61-4104-8112-2196ccb80fe7\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " Nov 02 19:49:24.322483 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.322377 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-openshift-ca\") pod \"37bb2374-6d61-4104-8112-2196ccb80fe7\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " Nov 02 19:49:24.322601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.322580 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-xks-ca\") pod \"37bb2374-6d61-4104-8112-2196ccb80fe7\" (UID: \"37bb2374-6d61-4104-8112-2196ccb80fe7\") " Nov 02 19:49:24.322848 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.322826 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-openshift-ca" pod "37bb2374-6d61-4104-8112-2196ccb80fe7" (UID: "37bb2374-6d61-4104-8112-2196ccb80fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:24.323422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.323395 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:24.324277 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.324256 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bb2374-6d61-4104-8112-2196ccb80fe7-kube-api-access-bwl4b" pod "37bb2374-6d61-4104-8112-2196ccb80fe7" (UID: "37bb2374-6d61-4104-8112-2196ccb80fe7"). InnerVolumeSpecName "kube-api-access-bwl4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:24.325226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.325206 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-xks-ca" pod "37bb2374-6d61-4104-8112-2196ccb80fe7" (UID: "37bb2374-6d61-4104-8112-2196ccb80fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:24.425291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.425253 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwl4b\" (UniqueName: \"kubernetes.io/projected/37bb2374-6d61-4104-8112-2196ccb80fe7-kube-api-access-bwl4b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:24.425465 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.425310 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bb2374-6d61-4104-8112-2196ccb80fe7-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:24.595829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.595743 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 19:49:24.595829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.595798 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 19:49:24.698449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.698408 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7137-30af7c"] Nov 02 19:49:24.706707 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.706672 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7137-30af7c"] Nov 02 19:49:24.755677 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.755633 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7137-30af7c" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" containerName="test-curl-7137-30af7c" containerID="cri-o://50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392" gracePeriod=2 Nov 02 19:49:24.756083 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.756061 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" Nov 02 19:49:24.786366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.759371 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm" event={"ID":"37bb2374-6d61-4104-8112-2196ccb80fe7","Type":"ContainerDied","Data":"5eeb98cd4af36c38d9b50188dfc32abd656ab8886cfcac2ea9c4db7d6429348a"} Nov 02 19:49:24.786366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.759475 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eeb98cd4af36c38d9b50188dfc32abd656ab8886cfcac2ea9c4db7d6429348a" Nov 02 19:49:24.806776 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:24.806648 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-30af7c\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.875789 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7137-b1d914"] Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876744 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8f20784f-9892-49f3-b784-c2837fa2afdf" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876765 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f20784f-9892-49f3-b784-c2837fa2afdf" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876791 4966 cpu_manager.go:414] "containerMap: removing container" podUID="37bb2374-6d61-4104-8112-2196ccb80fe7" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876800 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bb2374-6d61-4104-8112-2196ccb80fe7" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876820 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c970045e-af95-4c6b-be41-065f4837caaa" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876827 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c970045e-af95-4c6b-be41-065f4837caaa" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876868 4966 cpu_manager.go:414] "containerMap: removing container" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" containerName="test-curl-7137-30af7c" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.876875 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" containerName="test-curl-7137-30af7c" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.877035 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" containerName="test-curl-7137-30af7c" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.877055 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f20784f-9892-49f3-b784-c2837fa2afdf" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.877070 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bb2374-6d61-4104-8112-2196ccb80fe7" containerName="revoke-keys" Nov 02 19:49:24.883248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.877081 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c970045e-af95-4c6b-be41-065f4837caaa" containerName="revoke-keys" Nov 02 19:49:24.885617 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.885593 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:24.904525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:24.902684 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-b1d914"] Nov 02 19:49:24.914317 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:24.914266 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-30af7c\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:25.033237 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.033206 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm8rs\" (UniqueName: \"kubernetes.io/projected/bbc15a86-14fb-4d36-8f99-7696fe8d5eff-kube-api-access-gm8rs\") pod \"test-curl-7137-b1d914\" (UID: \"bbc15a86-14fb-4d36-8f99-7696fe8d5eff\") " pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:25.068741 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:25.068610 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-30af7c\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:25.134532 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.134449 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8rs\" (UniqueName: \"kubernetes.io/projected/bbc15a86-14fb-4d36-8f99-7696fe8d5eff-kube-api-access-gm8rs\") pod \"test-curl-7137-b1d914\" (UID: \"bbc15a86-14fb-4d36-8f99-7696fe8d5eff\") " pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:25.188320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.187821 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8rs\" (UniqueName: \"kubernetes.io/projected/bbc15a86-14fb-4d36-8f99-7696fe8d5eff-kube-api-access-gm8rs\") pod \"test-curl-7137-b1d914\" (UID: \"bbc15a86-14fb-4d36-8f99-7696fe8d5eff\") " pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:25.204404 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.204376 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:25.595267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.595206 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 19:49:25.595486 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.595279 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 19:49:25.634481 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:25.634453 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc15a86_14fb_4d36_8f99_7696fe8d5eff.slice/crio-1d5261003a1ddf3bc61b2817ba3dfe5f5192ef155ca0f498eb16061198588e75 WatchSource:0}: Error finding container 1d5261003a1ddf3bc61b2817ba3dfe5f5192ef155ca0f498eb16061198588e75: Status 404 returned error can't find the container with id 1d5261003a1ddf3bc61b2817ba3dfe5f5192ef155ca0f498eb16061198588e75 Nov 02 19:49:25.635468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.635444 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-b1d914"] Nov 02 19:49:25.780085 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:25.780046 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-b1d914" event={"ID":"bbc15a86-14fb-4d36-8f99-7696fe8d5eff","Type":"ContainerStarted","Data":"1d5261003a1ddf3bc61b2817ba3dfe5f5192ef155ca0f498eb16061198588e75"} Nov 02 19:49:26.596141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.596103 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 19:49:26.596508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.596470 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 19:49:26.729833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.729793 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj"] Nov 02 19:49:26.734421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.734397 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.797051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.797014 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj"] Nov 02 19:49:26.806690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.800599 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-b1d914" event={"ID":"bbc15a86-14fb-4d36-8f99-7696fe8d5eff","Type":"ContainerStarted","Data":"810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f"} Nov 02 19:49:26.806690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.802579 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.806690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.802721 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxh2\" (UniqueName: \"kubernetes.io/projected/cb668698-f97a-4c99-84ed-11c256d34fdf-kube-api-access-hpxh2\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.806690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.802747 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.828348 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.828322 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:49:26.828447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.828432 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:49:26.828548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.828535 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:49:26.829152 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.829135 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:49:26.903987 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.903899 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxh2\" (UniqueName: \"kubernetes.io/projected/cb668698-f97a-4c99-84ed-11c256d34fdf-kube-api-access-hpxh2\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.903987 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.903941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.904260 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.904054 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.904760 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.904739 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.907438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.907414 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:26.966088 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:26.965920 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxh2\" (UniqueName: \"kubernetes.io/projected/cb668698-f97a-4c99-84ed-11c256d34fdf-kube-api-access-hpxh2\") pod \"maas-api-revoke-sub-3c7a7495-qk6rj\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:27.000915 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.000851 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7137-b1d914" podStartSLOduration=3.000830347 podStartE2EDuration="3.000830347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:26.993434928 +0000 UTC m=+1272.942090767" watchObservedRunningTime="2026-11-02 19:49:27.000830347 +0000 UTC m=+1272.949486182" Nov 02 19:49:27.069879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.069849 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:27.108885 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.108856 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:27.208198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.208165 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0053b615-e806-46bd-9522-a6e7f3c39c8d-kube-api-access-bz94v\" (UniqueName: \"kubernetes.io/projected/0053b615-e806-46bd-9522-a6e7f3c39c8d-kube-api-access-bz94v\") pod \"0053b615-e806-46bd-9522-a6e7f3c39c8d\" (UID: \"0053b615-e806-46bd-9522-a6e7f3c39c8d\") " Nov 02 19:49:27.210840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.210811 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0053b615-e806-46bd-9522-a6e7f3c39c8d-kube-api-access-bz94v" pod "0053b615-e806-46bd-9522-a6e7f3c39c8d" (UID: "0053b615-e806-46bd-9522-a6e7f3c39c8d"). InnerVolumeSpecName "kube-api-access-bz94v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:27.310971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.310931 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz94v\" (UniqueName: \"kubernetes.io/projected/0053b615-e806-46bd-9522-a6e7f3c39c8d-kube-api-access-bz94v\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:27.478187 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.478141 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj"] Nov 02 19:49:27.484109 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:27.484073 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb668698_f97a_4c99_84ed_11c256d34fdf.slice/crio-c9e53335b9b1836f49bc48473dbbfbf202df4b2b88e3cfe98b6217984d105144 WatchSource:0}: Error finding container c9e53335b9b1836f49bc48473dbbfbf202df4b2b88e3cfe98b6217984d105144: Status 404 returned error can't find the container with id c9e53335b9b1836f49bc48473dbbfbf202df4b2b88e3cfe98b6217984d105144 Nov 02 19:49:27.595233 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.595194 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 02 19:49:27.595427 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.595276 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 02 19:49:27.813004 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.812922 4966 generic.go:356] "Generic (PLEG): container finished" podID="0053b615-e806-46bd-9522-a6e7f3c39c8d" containerID="50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392" exitCode=137 Nov 02 19:49:27.813430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.813007 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:27.813430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.813034 4966 scope.go:122] "RemoveContainer" containerID="50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392" Nov 02 19:49:27.816854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.815737 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" event={"ID":"cb668698-f97a-4c99-84ed-11c256d34fdf","Type":"ContainerStarted","Data":"c9e53335b9b1836f49bc48473dbbfbf202df4b2b88e3cfe98b6217984d105144"} Nov 02 19:49:27.828609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.828586 4966 scope.go:122] "RemoveContainer" containerID="50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392" Nov 02 19:49:27.828885 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:27.828869 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392\": container with ID starting with 50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392 not found: ID does not exist" containerID="50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392" Nov 02 19:49:27.828929 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:27.828894 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392"} err="failed to get container status \"50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392\": rpc error: code = NotFound desc = could not find container \"50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392\": container with ID starting with 50116736eace7828a656a3dbbb1b9381e169ce921d11aaeb7082942fac03a392 not found: ID does not exist" Nov 02 19:49:27.872507 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:27.872450 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-30af7c\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" pod="opendatahub/test-curl-7137-30af7c" Nov 02 19:49:28.184873 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:28.184796 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0053b615-e806-46bd-9522-a6e7f3c39c8d" path="/var/lib/kubelet/pods/0053b615-e806-46bd-9522-a6e7f3c39c8d/volumes" Nov 02 19:49:28.607783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:28.592205 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7137-b1d914"] Nov 02 19:49:28.607783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:28.604597 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7137-b1d914"] Nov 02 19:49:28.830872 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:28.830830 4966 generic.go:356] "Generic (PLEG): container finished" podID="cb668698-f97a-4c99-84ed-11c256d34fdf" containerID="a52743a0936882429dcc7add2b5f8508bc18b95163e53b1b04cf30639ff104ea" exitCode=0 Nov 02 19:49:28.831289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:28.830935 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" event={"ID":"cb668698-f97a-4c99-84ed-11c256d34fdf","Type":"ContainerDied","Data":"a52743a0936882429dcc7add2b5f8508bc18b95163e53b1b04cf30639ff104ea"} Nov 02 19:49:28.831289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:28.831222 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7137-b1d914" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" containerName="test-curl-7137-b1d914" containerID="cri-o://810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f" gracePeriod=2 Nov 02 19:49:28.993783 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:28.993738 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-b1d914\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:29.026360 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:29.026327 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:29.026691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:29.026666 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:29.027449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:29.027429 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:49:29.186683 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:29.186573 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-b1d914\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:29.344226 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:29.344178 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-b1d914\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:30.231833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.231805 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:30.257123 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.257079 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-openshift-ca\") pod \"cb668698-f97a-4c99-84ed-11c256d34fdf\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " Nov 02 19:49:30.257282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.257197 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb668698-f97a-4c99-84ed-11c256d34fdf-kube-api-access-hpxh2\" (UniqueName: \"kubernetes.io/projected/cb668698-f97a-4c99-84ed-11c256d34fdf-kube-api-access-hpxh2\") pod \"cb668698-f97a-4c99-84ed-11c256d34fdf\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " Nov 02 19:49:30.257403 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.257370 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-xks-ca\") pod \"cb668698-f97a-4c99-84ed-11c256d34fdf\" (UID: \"cb668698-f97a-4c99-84ed-11c256d34fdf\") " Nov 02 19:49:30.258214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.258186 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-openshift-ca" pod "cb668698-f97a-4c99-84ed-11c256d34fdf" (UID: "cb668698-f97a-4c99-84ed-11c256d34fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:30.259695 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.259662 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb668698-f97a-4c99-84ed-11c256d34fdf-kube-api-access-hpxh2" pod "cb668698-f97a-4c99-84ed-11c256d34fdf" (UID: "cb668698-f97a-4c99-84ed-11c256d34fdf"). InnerVolumeSpecName "kube-api-access-hpxh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:30.259926 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.259908 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-xks-ca" pod "cb668698-f97a-4c99-84ed-11c256d34fdf" (UID: "cb668698-f97a-4c99-84ed-11c256d34fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:30.407357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.406476 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:30.407357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.406530 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb668698-f97a-4c99-84ed-11c256d34fdf-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:30.407357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.406571 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxh2\" (UniqueName: \"kubernetes.io/projected/cb668698-f97a-4c99-84ed-11c256d34fdf-kube-api-access-hpxh2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:30.861735 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.861710 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" Nov 02 19:49:30.861875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.861744 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj" event={"ID":"cb668698-f97a-4c99-84ed-11c256d34fdf","Type":"ContainerDied","Data":"c9e53335b9b1836f49bc48473dbbfbf202df4b2b88e3cfe98b6217984d105144"} Nov 02 19:49:30.861875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:30.861843 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9e53335b9b1836f49bc48473dbbfbf202df4b2b88e3cfe98b6217984d105144" Nov 02 19:49:31.168797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.168767 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:31.338315 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.326030 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc15a86-14fb-4d36-8f99-7696fe8d5eff-kube-api-access-gm8rs\" (UniqueName: \"kubernetes.io/projected/bbc15a86-14fb-4d36-8f99-7696fe8d5eff-kube-api-access-gm8rs\") pod \"bbc15a86-14fb-4d36-8f99-7696fe8d5eff\" (UID: \"bbc15a86-14fb-4d36-8f99-7696fe8d5eff\") " Nov 02 19:49:31.348574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.348522 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc15a86-14fb-4d36-8f99-7696fe8d5eff-kube-api-access-gm8rs" pod "bbc15a86-14fb-4d36-8f99-7696fe8d5eff" (UID: "bbc15a86-14fb-4d36-8f99-7696fe8d5eff"). InnerVolumeSpecName "kube-api-access-gm8rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:31.428645 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.428560 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm8rs\" (UniqueName: \"kubernetes.io/projected/bbc15a86-14fb-4d36-8f99-7696fe8d5eff-kube-api-access-gm8rs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:31.536604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.536570 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh"] Nov 02 19:49:31.537450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.537418 4966 cpu_manager.go:414] "containerMap: removing container" podUID="cb668698-f97a-4c99-84ed-11c256d34fdf" containerName="revoke-keys" Nov 02 19:49:31.537450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.537440 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb668698-f97a-4c99-84ed-11c256d34fdf" containerName="revoke-keys" Nov 02 19:49:31.537654 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.537475 4966 cpu_manager.go:414] "containerMap: removing container" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" containerName="test-curl-7137-b1d914" Nov 02 19:49:31.537654 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.537482 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" containerName="test-curl-7137-b1d914" Nov 02 19:49:31.537748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.537654 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb668698-f97a-4c99-84ed-11c256d34fdf" containerName="revoke-keys" Nov 02 19:49:31.537748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.537672 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" containerName="test-curl-7137-b1d914" Nov 02 19:49:31.542154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.542130 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.612344 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.612313 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d23a1e27-dockercfg-4z4q7\"" Nov 02 19:49:31.612519 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.612408 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d23a1e27\"" Nov 02 19:49:31.623373 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.619827 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh"] Nov 02 19:49:31.663708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.663651 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skx7\" (UniqueName: \"kubernetes.io/projected/c0ee65ed-8d42-43a1-837e-72f14e451d43-kube-api-access-8skx7\") pod \"payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.664459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.664345 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee65ed-8d42-43a1-837e-72f14e451d43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.732472 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.707000 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng"] Nov 02 19:49:31.732472 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.719580 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:31.755226 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.755186 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng"] Nov 02 19:49:31.766850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.766404 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skx7\" (UniqueName: \"kubernetes.io/projected/c0ee65ed-8d42-43a1-837e-72f14e451d43-kube-api-access-8skx7\") pod \"payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.767281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.767238 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee65ed-8d42-43a1-837e-72f14e451d43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.768272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.768246 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee65ed-8d42-43a1-837e-72f14e451d43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.860321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.860223 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skx7\" (UniqueName: \"kubernetes.io/projected/c0ee65ed-8d42-43a1-837e-72f14e451d43-kube-api-access-8skx7\") pod \"payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.862343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.860600 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:31.876951 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.876906 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhvg\" (UniqueName: \"kubernetes.io/projected/9c763680-0ea9-480e-8b0b-c6fe9b800529-kube-api-access-klhvg\") pod \"payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:31.877115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.877026 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c763680-0ea9-480e-8b0b-c6fe9b800529-plugins-config-volume\") pod \"payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:31.941131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.940998 4966 generic.go:356] "Generic (PLEG): container finished" podID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" containerID="810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f" exitCode=137 Nov 02 19:49:31.941131 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.941090 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:31.941351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.941161 4966 scope.go:122] "RemoveContainer" containerID="810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f" Nov 02 19:49:31.974074 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.971986 4966 scope.go:122] "RemoveContainer" containerID="810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f" Nov 02 19:49:31.976456 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:31.975370 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f\": container with ID starting with 810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f not found: ID does not exist" containerID="810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f" Nov 02 19:49:31.976456 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.975417 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f"} err="failed to get container status \"810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f\": rpc error: code = NotFound desc = could not find container \"810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f\": container with ID starting with 810035723adaa47c2f362687ab390e06b22a092437e9307d9ca88d4df2bb8d4f not found: ID does not exist" Nov 02 19:49:31.987194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.986638 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhvg\" (UniqueName: \"kubernetes.io/projected/9c763680-0ea9-480e-8b0b-c6fe9b800529-kube-api-access-klhvg\") pod \"payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:31.987194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.986767 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c763680-0ea9-480e-8b0b-c6fe9b800529-plugins-config-volume\") pod \"payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:31.991692 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:31.987889 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c763680-0ea9-480e-8b0b-c6fe9b800529-plugins-config-volume\") pod \"payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:32.014620 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:32.014543 4966 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-b1d914\" is forbidden: User \"system:node:ip-10-0-132-174.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-174.us-west-2.compute.internal' and this object" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" pod="opendatahub/test-curl-7137-b1d914" Nov 02 19:49:32.042336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.042200 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhvg\" (UniqueName: \"kubernetes.io/projected/9c763680-0ea9-480e-8b0b-c6fe9b800529-kube-api-access-klhvg\") pod \"payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:32.087321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.083603 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:32.201605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.201570 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc15a86-14fb-4d36-8f99-7696fe8d5eff" path="/var/lib/kubelet/pods/bbc15a86-14fb-4d36-8f99-7696fe8d5eff/volumes" Nov 02 19:49:32.340960 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:32.340927 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ee65ed_8d42_43a1_837e_72f14e451d43.slice/crio-bbeeecdb3cf7baabd3980e07800b4a2670f6a075b63ab23786b529d1026ea6b4 WatchSource:0}: Error finding container bbeeecdb3cf7baabd3980e07800b4a2670f6a075b63ab23786b529d1026ea6b4: Status 404 returned error can't find the container with id bbeeecdb3cf7baabd3980e07800b4a2670f6a075b63ab23786b529d1026ea6b4 Nov 02 19:49:32.343213 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.343189 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh"] Nov 02 19:49:32.505834 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:32.505789 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c763680_0ea9_480e_8b0b_c6fe9b800529.slice/crio-35f19a1afcca14e956f96c107042ac04df39685fd39a990bd92aa48d1e719745 WatchSource:0}: Error finding container 35f19a1afcca14e956f96c107042ac04df39685fd39a990bd92aa48d1e719745: Status 404 returned error can't find the container with id 35f19a1afcca14e956f96c107042ac04df39685fd39a990bd92aa48d1e719745 Nov 02 19:49:32.507680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.507549 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng"] Nov 02 19:49:32.953598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.953561 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" event={"ID":"9c763680-0ea9-480e-8b0b-c6fe9b800529","Type":"ContainerStarted","Data":"336ba983273504b3fedbf6d4fb1961a166515149a3d74e9305e9489cf3b7fc7e"} Nov 02 19:49:32.953815 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.953624 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" event={"ID":"9c763680-0ea9-480e-8b0b-c6fe9b800529","Type":"ContainerStarted","Data":"35f19a1afcca14e956f96c107042ac04df39685fd39a990bd92aa48d1e719745"} Nov 02 19:49:32.953815 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.953707 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:32.955746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.955716 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" event={"ID":"c0ee65ed-8d42-43a1-837e-72f14e451d43","Type":"ContainerStarted","Data":"753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020"} Nov 02 19:49:32.955888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.955768 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" event={"ID":"c0ee65ed-8d42-43a1-837e-72f14e451d43","Type":"ContainerStarted","Data":"bbeeecdb3cf7baabd3980e07800b4a2670f6a075b63ab23786b529d1026ea6b4"} Nov 02 19:49:32.955888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:32.955835 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:33.062832 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:33.062764 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" podStartSLOduration=2.062744106 podStartE2EDuration="2.062744106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:33.058856026 +0000 UTC m=+1279.007511866" watchObservedRunningTime="2026-11-02 19:49:33.062744106 +0000 UTC m=+1279.011399946" Nov 02 19:49:33.170446 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:33.170381 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" podStartSLOduration=2.170362183 podStartE2EDuration="2.170362183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:33.163460366 +0000 UTC m=+1279.112116206" watchObservedRunningTime="2026-11-02 19:49:33.170362183 +0000 UTC m=+1279.119018023" Nov 02 19:49:34.831645 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.831609 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8"] Nov 02 19:49:34.836088 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.836064 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.879114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.878866 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56"] Nov 02 19:49:34.883992 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.883967 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.931970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.931938 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8"] Nov 02 19:49:34.951755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.951718 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.951886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.951772 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.951886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.951864 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.951993 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.951940 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.952085 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952066 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.952136 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952101 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.952236 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952214 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.952279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952264 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.952365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952342 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.952446 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952430 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvt9\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-kube-api-access-qwvt9\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.952491 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952459 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.952527 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952493 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.952565 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952524 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.952614 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952567 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.952681 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952664 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.952758 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952743 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqwf\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-kube-api-access-grqwf\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.952813 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952779 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:34.952888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.952866 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:34.983866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.982370 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz"] Nov 02 19:49:34.990124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.989917 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:34.990124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:34.990076 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56"] Nov 02 19:49:35.014436 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.011634 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth"] Nov 02 19:49:35.019097 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.019073 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.055319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055260 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.055508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055365 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.055508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055447 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.055508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055499 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62aa0383-eacb-4759-9e8d-9535b20289c7-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.055762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055533 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqwf\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-kube-api-access-grqwf\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.055762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055570 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.055762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055632 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.055762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055685 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.055762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055709 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.055762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055741 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055773 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kzxm\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-kube-api-access-4kzxm\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055825 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055875 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055905 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.055975 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056010 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.056054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056037 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056071 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056162 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056198 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056244 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056275 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056363 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvt9\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-kube-api-access-qwvt9\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056392 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056431 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.056793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.056467 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.057318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.057040 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.057943 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.057913 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.058338 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.058268 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-certs\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.060070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.058700 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.060070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.058932 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-data\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.060070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.059106 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-credential-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.060070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.059516 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.060070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.059623 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.060070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.060031 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istiod-ca-cert\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.060858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.060835 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-socket\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.063910 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.063629 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.066430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.066399 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.066508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.066426 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-podinfo\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.067741 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.067719 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-envoy\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.068635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.068615 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz"] Nov 02 19:49:35.099803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.099706 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth"] Nov 02 19:49:35.112588 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.112546 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.112799 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.112772 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvt9\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-kube-api-access-qwvt9\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.113369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.113342 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqwf\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-kube-api-access-grqwf\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.114271 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.114243 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-token\") pod \"e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.114789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.114760 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl"] Nov 02 19:49:35.121824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.121803 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.153950 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.153929 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:35.157768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.157612 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.157768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.157690 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.157921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.157782 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6tdn\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-kube-api-access-l6tdn\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.157921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.157831 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.157932 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.158021 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.157987 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62aa0383-eacb-4759-9e8d-9535b20289c7-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158021 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.158127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158074 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5792127-6974-484d-8ca2-8cb8962664ef-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.158127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158100 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158128 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.158265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158161 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158190 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158216 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kzxm\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-kube-api-access-4kzxm\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158270 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158341 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.158504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158372 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.158504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158397 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.158504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.158413 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5792127-6974-484d-8ca2-8cb8962664ef-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.159375 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.159354 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62aa0383-eacb-4759-9e8d-9535b20289c7-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.159669 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.159644 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.159932 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.159913 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.161357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.160168 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.161357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.161000 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.162229 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.161989 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.163111 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.163085 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.163601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.163583 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl"] Nov 02 19:49:35.170407 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.170385 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:49:35.170517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.170393 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:49:35.170517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.170481 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:49:35.170638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.170590 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:49:35.200788 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.199336 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:35.212005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.211976 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.212859 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.212834 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kzxm\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-kube-api-access-4kzxm\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.260340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.260261 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.260340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.260327 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.260575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.260355 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5792127-6974-484d-8ca2-8cb8962664ef-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.260575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.260462 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.260575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.260504 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6tdn\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-kube-api-access-l6tdn\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.260575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.260546 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.262078 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.262050 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-data\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.263565 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.263510 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-envoy\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.265494 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.265466 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-credential-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.265961 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.265942 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5792127-6974-484d-8ca2-8cb8962664ef-istio-podinfo\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.266466 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.260629 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.266536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.266519 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.266594 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.266581 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5792127-6974-484d-8ca2-8cb8962664ef-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.266637 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.266624 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.266671 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.266658 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.266736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.266723 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ch9m\" (UniqueName: \"kubernetes.io/projected/3343608e-2dbd-4473-a794-0e96a7169300-kube-api-access-9ch9m\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.267140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.267121 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-certs\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.267773 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.267753 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5792127-6974-484d-8ca2-8cb8962664ef-istiod-ca-cert\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.268038 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.268020 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-socket\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.319212 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.319175 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:35.321436 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.321407 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6tdn\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-kube-api-access-l6tdn\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.330622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.330595 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-istio-token\") pod \"e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.336514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.336489 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:35.368517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.368393 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.368625 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.368601 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.368694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.368672 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ch9m\" (UniqueName: \"kubernetes.io/projected/3343608e-2dbd-4473-a794-0e96a7169300-kube-api-access-9ch9m\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.369856 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.369794 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.384919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.384885 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.433241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.427713 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ch9m\" (UniqueName: \"kubernetes.io/projected/3343608e-2dbd-4473-a794-0e96a7169300-kube-api-access-9ch9m\") pod \"maas-api-revoke-sub-1863c658-ngrgl\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.454630 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.454180 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:35.576217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.576185 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8"] Nov 02 19:49:35.583231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.583195 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.583367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.583276 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.583367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.583335 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.609553 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.609497 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56"] Nov 02 19:49:35.611793 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:35.611759 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2bf8e9_8ac7_4b1b_b057_68cb0b57a69b.slice/crio-f0814f44085ba7dc694e61bd6ae4fcbcb08b045d5fde82ef115d2ac783d9e151 WatchSource:0}: Error finding container f0814f44085ba7dc694e61bd6ae4fcbcb08b045d5fde82ef115d2ac783d9e151: Status 404 returned error can't find the container with id f0814f44085ba7dc694e61bd6ae4fcbcb08b045d5fde82ef115d2ac783d9e151 Nov 02 19:49:35.620105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.619568 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.620105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.619654 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.620105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.619697 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.748542 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.748018 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.748542 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.748101 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.748542 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.748142 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.761603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.752632 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz"] Nov 02 19:49:35.778345 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:35.776702 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5792127_6974_484d_8ca2_8cb8962664ef.slice/crio-69e3e4ca640749347209615a87a725a64735078b58d6ab5da115e9b6c5ba2627 WatchSource:0}: Error finding container 69e3e4ca640749347209615a87a725a64735078b58d6ab5da115e9b6c5ba2627: Status 404 returned error can't find the container with id 69e3e4ca640749347209615a87a725a64735078b58d6ab5da115e9b6c5ba2627 Nov 02 19:49:35.781013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.780082 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth"] Nov 02 19:49:35.781013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.780207 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.781013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.780278 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.781013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.780359 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:49:35.890538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:35.889103 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl"] Nov 02 19:49:35.897162 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:35.896741 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3343608e_2dbd_4473_a794_0e96a7169300.slice/crio-a6f07c4f46759ac3c05b1be7e88602813ae2464a06883b1a1595c1a66db00674 WatchSource:0}: Error finding container a6f07c4f46759ac3c05b1be7e88602813ae2464a06883b1a1595c1a66db00674: Status 404 returned error can't find the container with id a6f07c4f46759ac3c05b1be7e88602813ae2464a06883b1a1595c1a66db00674 Nov 02 19:49:36.031322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.029155 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" event={"ID":"a75f0bed-989b-437c-a99f-85eb67f5cb0c","Type":"ContainerStarted","Data":"b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813"} Nov 02 19:49:36.031322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.029229 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" event={"ID":"a75f0bed-989b-437c-a99f-85eb67f5cb0c","Type":"ContainerStarted","Data":"dac25b0eee7cc3aa041f1fe8adef1726f9c6f68786b02606599058d1b6b6585f"} Nov 02 19:49:36.033352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.032682 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" event={"ID":"3343608e-2dbd-4473-a794-0e96a7169300","Type":"ContainerStarted","Data":"a6f07c4f46759ac3c05b1be7e88602813ae2464a06883b1a1595c1a66db00674"} Nov 02 19:49:36.036269 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.036239 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" event={"ID":"62aa0383-eacb-4759-9e8d-9535b20289c7","Type":"ContainerStarted","Data":"cd4633fcca89dc7eaf1c0fb7d1dcf4083734f289510d5f1921e5afa5a39f3a61"} Nov 02 19:49:36.036381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.036312 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" event={"ID":"62aa0383-eacb-4759-9e8d-9535b20289c7","Type":"ContainerStarted","Data":"bb4de03f78e36a96d8c51dd9cde77528201b5c0a0b628831a0899acbc76fbd3d"} Nov 02 19:49:36.040377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.039780 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" event={"ID":"b5792127-6974-484d-8ca2-8cb8962664ef","Type":"ContainerStarted","Data":"62fdb37fb509f64f6b1ee701ebbc7e3244a83d9f519a7a39a4582306f61908fb"} Nov 02 19:49:36.040377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.039827 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" event={"ID":"b5792127-6974-484d-8ca2-8cb8962664ef","Type":"ContainerStarted","Data":"69e3e4ca640749347209615a87a725a64735078b58d6ab5da115e9b6c5ba2627"} Nov 02 19:49:36.043551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.042876 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" event={"ID":"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b","Type":"ContainerStarted","Data":"bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f"} Nov 02 19:49:36.043551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.042922 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" event={"ID":"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b","Type":"ContainerStarted","Data":"f0814f44085ba7dc694e61bd6ae4fcbcb08b045d5fde82ef115d2ac783d9e151"} Nov 02 19:49:36.163028 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.161110 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:36.167339 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.166623 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podStartSLOduration=2.166601585 podStartE2EDuration="2.166601585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:36.158628518 +0000 UTC m=+1282.107284350" watchObservedRunningTime="2026-11-02 19:49:36.166601585 +0000 UTC m=+1282.115257425" Nov 02 19:49:36.169357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.168403 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:36.169357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.168461 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:36.201349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.201287 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:36.201532 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.201442 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:36.201532 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.201486 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:36.300319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.293235 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podStartSLOduration=2.293210905 podStartE2EDuration="2.293210905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:36.281536564 +0000 UTC m=+1282.230192440" watchObservedRunningTime="2026-11-02 19:49:36.293210905 +0000 UTC m=+1282.241866745" Nov 02 19:49:36.319699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.319665 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:36.321214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.321183 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:36.321369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.321245 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:36.336929 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.336895 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:36.352818 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.352787 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:36.353058 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.353026 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:36.402194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.402118 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podStartSLOduration=2.40209395 podStartE2EDuration="2.40209395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:36.391012557 +0000 UTC m=+1282.339668401" watchObservedRunningTime="2026-11-02 19:49:36.40209395 +0000 UTC m=+1282.350749793" Nov 02 19:49:36.521321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:36.519834 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podStartSLOduration=2.519810164 podStartE2EDuration="2.519810164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:36.511564062 +0000 UTC m=+1282.460219903" watchObservedRunningTime="2026-11-02 19:49:36.519810164 +0000 UTC m=+1282.468466007" Nov 02 19:49:37.071509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.069582 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" event={"ID":"3343608e-2dbd-4473-a794-0e96a7169300","Type":"ContainerStarted","Data":"75d04b7d7d1a212e7daf4c926755141aed1015e88c8a013292f6ac47aa056a3c"} Nov 02 19:49:37.158888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.157732 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:37.158888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.157791 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:37.200181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.200024 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:37.200181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.200115 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:37.215452 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.202016 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" podStartSLOduration=2.201996612 podStartE2EDuration="2.201996612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:37.199710001 +0000 UTC m=+1283.148365843" watchObservedRunningTime="2026-11-02 19:49:37.201996612 +0000 UTC m=+1283.150652453" Nov 02 19:49:37.324428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.320847 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:37.324428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.320910 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:37.347130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.347058 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:37.347460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.347421 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:37.644073 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:37.643959 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:38.176325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.173638 4966 generic.go:356] "Generic (PLEG): container finished" podID="3343608e-2dbd-4473-a794-0e96a7169300" containerID="75d04b7d7d1a212e7daf4c926755141aed1015e88c8a013292f6ac47aa056a3c" exitCode=0 Nov 02 19:49:38.201322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.194996 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" event={"ID":"3343608e-2dbd-4473-a794-0e96a7169300","Type":"ContainerDied","Data":"75d04b7d7d1a212e7daf4c926755141aed1015e88c8a013292f6ac47aa056a3c"} Nov 02 19:49:38.218349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.213469 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:38.218349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.213548 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:38.218349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.216226 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:38.218349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.216273 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:38.302645 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.302582 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:38.303144 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.303109 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 19:49:38.319647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.319603 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:38.346204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.346134 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:38.346479 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.346446 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:38.411531 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.378892 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:38.481765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.476940 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:38.592324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.568263 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:38.592324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.578808 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 19:49:38.654939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.616535 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:38.654939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.650575 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 19:49:38.654939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.652621 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 19:49:38.671324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.668223 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:49:38.722398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:38.720781 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.151530 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.151417 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.163894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.163776 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:39.163894 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.163833 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:39.244322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.216671 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:39.244322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.216735 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:39.356321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.310714 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.439678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.439583 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:49:39.440326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.356579 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.441125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.357912 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 19:49:39.441435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.385225 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.441735 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.441719 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:49:39.441905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.441889 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:49:39.442331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.385346 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:39.442485 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.442460 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:39.518413 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.400849 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 19:49:39.590974 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.590925 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.617171 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.616158 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.624695 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.623348 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.651451 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.625866 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:49:39.708514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.708421 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:49:39.708786 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.708766 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:49:39.748325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.647575 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.775938 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.526009 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 19:49:39.776647 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.675156 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:39.776868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.776838 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:39.776979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.659884 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:39.785646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.650961 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.168959 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.159321 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:40.169280 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.169235 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:40.211873 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:39.762664 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.211873 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.210980 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:49:40.222779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.222744 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:40.223031 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.222998 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:40.369322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.355449 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:40.369322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.355508 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:40.369322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.355957 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:40.369322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.355993 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:40.446400 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.373258 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:49:40.450322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.447210 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.495915 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.495874 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:49:40.496650 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.496600 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.497216 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.497177 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 19:49:40.497729 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.497710 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:49:40.571721 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.571655 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.628975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.628754 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 19:49:40.628975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.628861 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.628975 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.628905 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.660421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.658151 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:49:40.816005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.760067 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:49:40.817522 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.817474 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:40.817782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:40.817750 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:49:41.161514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.161419 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.285908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.285857 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.288138 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.288095 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.301136 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.311821 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.328459 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.321009 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.328551 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.339099 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.339141 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:41.376331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.326886 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:41.461824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.461719 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 19:49:41.479376 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.464197 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:49:41.520986 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.499734 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 19:49:41.529552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.515442 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.529552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.515497 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:49:41.540371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.535083 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:41.666325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.658441 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.666325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.658901 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.666325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.659288 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.852325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.832153 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 19:49:41.855477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.854965 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:41.860191 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.857810 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-xks-ca\") pod \"3343608e-2dbd-4473-a794-0e96a7169300\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " Nov 02 19:49:41.860191 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.857923 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-openshift-ca\") pod \"3343608e-2dbd-4473-a794-0e96a7169300\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " Nov 02 19:49:41.860191 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.857973 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3343608e-2dbd-4473-a794-0e96a7169300-kube-api-access-9ch9m\" (UniqueName: \"kubernetes.io/projected/3343608e-2dbd-4473-a794-0e96a7169300-kube-api-access-9ch9m\") pod \"3343608e-2dbd-4473-a794-0e96a7169300\" (UID: \"3343608e-2dbd-4473-a794-0e96a7169300\") " Nov 02 19:49:41.888326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.887676 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:49:41.978320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:41.952946 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-openshift-ca" pod "3343608e-2dbd-4473-a794-0e96a7169300" (UID: "3343608e-2dbd-4473-a794-0e96a7169300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:42.039536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.026613 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/health\": context deadline exceeded" Nov 02 19:49:42.039536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.026924 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3343608e-2dbd-4473-a794-0e96a7169300-kube-api-access-9ch9m" pod "3343608e-2dbd-4473-a794-0e96a7169300" (UID: "3343608e-2dbd-4473-a794-0e96a7169300"). InnerVolumeSpecName "kube-api-access-9ch9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:42.135370 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.100592 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:42.135370 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.100637 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ch9m\" (UniqueName: \"kubernetes.io/projected/3343608e-2dbd-4473-a794-0e96a7169300-kube-api-access-9ch9m\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:42.155853 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.155814 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:42.156133 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.156104 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:42.195345 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.182181 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-xks-ca" pod "3343608e-2dbd-4473-a794-0e96a7169300" (UID: "3343608e-2dbd-4473-a794-0e96a7169300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:42.399525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.313640 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:42.422321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.327723 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:42.422321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.383109 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:42.422321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.418681 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:42.422321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.383564 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3343608e-2dbd-4473-a794-0e96a7169300-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:42.422321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.383174 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:42.422321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.418779 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:42.664384 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.555843 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:42.664384 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.555926 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:42.664384 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.610070 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:42.789359 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.788468 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:42.834178 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.834115 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:42.882470 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.882417 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:42.887319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.883258 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 19:49:42.917377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.891783 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 19:49:42.971279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.888691 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:42.972221 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.926409 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 02 19:49:42.983365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:42.978266 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:43.056352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.042424 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 19:49:43.293559 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.293506 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:49:43.390071 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.390035 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:43.390398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.390346 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:43.435349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.411909 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:43.435349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.411970 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:43.435349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.413466 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:43.435349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.413512 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:43.435349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.435307 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:43.436211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.436182 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 19:49:43.446502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.446471 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:43.446785 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.446741 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:43.641195 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:43.641111 4966 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.431s" Nov 02 19:49:43.641446 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.641418 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" event={"ID":"3343608e-2dbd-4473-a794-0e96a7169300","Type":"ContainerDied","Data":"a6f07c4f46759ac3c05b1be7e88602813ae2464a06883b1a1595c1a66db00674"} Nov 02 19:49:43.641605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.641589 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f07c4f46759ac3c05b1be7e88602813ae2464a06883b1a1595c1a66db00674" Nov 02 19:49:43.672559 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.667974 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl" Nov 02 19:49:43.719608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.697119 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:49:43.965371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.956038 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:43.965371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.957292 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:43.974845 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.974803 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:43.975411 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.975380 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 19:49:43.985774 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.983379 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:43.985774 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.983717 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:43.986876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:43.986855 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:49:44.029194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.007481 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:49:44.029194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.022994 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:49:44.083322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.064520 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:44.234500 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.234463 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:44.264881 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.264837 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:44.329337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.311621 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:44.329337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.311685 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:44.329337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.312859 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 19:49:44.329337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.314444 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:44.329337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.325208 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:44.329337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.325252 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:44.339391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.339252 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:44.339391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.339326 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:44.450693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.450624 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Nov 02 19:49:44.451217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.450649 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 19:49:44.961825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.961728 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:49:44.962494 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.962452 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 19:49:44.963877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.963839 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:44.985962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.985438 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:44.985962 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.985746 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 19:49:44.997574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.989617 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 02 19:49:44.997574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:44.989561 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.025199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.014469 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.072521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.072476 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.206322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.200928 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:45.206322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.200984 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:45.324322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.320469 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.357666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.356866 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:45.357666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.356930 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:45.357666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.357491 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:45.357666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.357533 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:45.359238 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.358142 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.481325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.466056 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:45.481325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.466131 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:45.483959 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.483922 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Nov 02 19:49:45.531312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.484137 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 19:49:45.535888 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:49:45.531602 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:49:45.966390 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.965344 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.986182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.985327 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.986182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.985679 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.986182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.985949 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.994321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.993211 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:45.994321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:45.993887 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 02 19:49:46.025503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.024867 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:49:46.037096 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.037066 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:49:46.040585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.040553 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 19:49:46.041094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.041061 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:49:46.062331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.047659 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:46.112780 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.112746 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7"] Nov 02 19:49:46.125447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.125418 4966 cpu_manager.go:414] "containerMap: removing container" podUID="3343608e-2dbd-4473-a794-0e96a7169300" containerName="revoke-keys" Nov 02 19:49:46.125657 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.125643 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3343608e-2dbd-4473-a794-0e96a7169300" containerName="revoke-keys" Nov 02 19:49:46.126195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.126175 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3343608e-2dbd-4473-a794-0e96a7169300" containerName="revoke-keys" Nov 02 19:49:46.151316 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.142951 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7"] Nov 02 19:49:46.151316 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.143035 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:49:46.151316 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.143444 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.183363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.169377 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:46.183363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.169432 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:46.200200 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.199584 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:49:46.200200 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.199778 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:49:46.200200 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.199887 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:49:46.219416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.213733 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:49:46.228207 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.227982 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:46.259336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.256753 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 19:49:46.259336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.256811 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 19:49:46.259819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.259793 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:49:46.260905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.260888 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:49:46.272129 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.272105 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:49:46.332164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.320854 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:49:46.332164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.321178 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:46.332164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.321221 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:46.332164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.327624 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.332164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.327720 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.332164 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.327881 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmxl\" (UniqueName: \"kubernetes.io/projected/92e01470-20f7-4ded-ad9b-63ec1848cf5d-kube-api-access-mxmxl\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.344330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.342699 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 19:49:46.344330 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.342748 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 19:49:46.365987 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.365960 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:49:46.374765 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.374739 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:49:46.421099 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.418795 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:49:46.436762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.431715 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.436762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.431835 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.436762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.432003 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmxl\" (UniqueName: \"kubernetes.io/projected/92e01470-20f7-4ded-ad9b-63ec1848cf5d-kube-api-access-mxmxl\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.436762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.433944 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.437966 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.437945 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:49:46.472447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.468756 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.521907 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.521692 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmxl\" (UniqueName: \"kubernetes.io/projected/92e01470-20f7-4ded-ad9b-63ec1848cf5d-kube-api-access-mxmxl\") pod \"maas-api-revoke-sub-cc4df6b7-t8zh7\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.531573 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.531212 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:46.624938 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:46.623287 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:49:47.159749 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.159617 4966 patch_prober.go:29] interesting pod/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 19:49:47.159749 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.159676 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 19:49:47.325375 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.320386 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 19:49:47.325375 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.320468 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 19:49:47.691022 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.668612 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:47.731362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.720269 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:47.740454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.740409 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7"] Nov 02 19:49:47.800575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.778729 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:49:47.999382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:47.999353 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:48.065323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.043766 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:48.086323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.079115 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:49:48.360151 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.360051 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" event={"ID":"92e01470-20f7-4ded-ad9b-63ec1848cf5d","Type":"ContainerStarted","Data":"a44de15711bc00d73a568d7380b9a2bf6e0f355697d1c97c9221497afacef593"} Nov 02 19:49:48.420357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.415423 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:48.420357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.416966 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:48.420617 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.420405 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:49:48.514722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.514685 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:48.515154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.515127 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:48.516487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.516456 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:49:48.575320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.574348 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq"] Nov 02 19:49:48.589868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.589835 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.607863 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.607831 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq"] Nov 02 19:49:48.608523 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.608481 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.608740 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.608710 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.609196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.609165 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6n82\" (UniqueName: \"kubernetes.io/projected/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-kube-api-access-q6n82\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.710221 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.710189 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n82\" (UniqueName: \"kubernetes.io/projected/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-kube-api-access-q6n82\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.710836 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.710349 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.710836 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.710387 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.711395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.711365 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.716081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.716054 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.805958 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.805171 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6n82\" (UniqueName: \"kubernetes.io/projected/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-kube-api-access-q6n82\") pod \"maas-api-revoke-sub-f421424a-fqpqq\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:48.931166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:48.931081 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:49.324967 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:49:49.324931 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ab6e1b_42b4_4b21_9012_4dbd1be85462.slice/crio-34ca10faef96f3c660977612e46a8bf3755a53e9dd13820c0efc7cdabc6bf1c3 WatchSource:0}: Error finding container 34ca10faef96f3c660977612e46a8bf3755a53e9dd13820c0efc7cdabc6bf1c3: Status 404 returned error can't find the container with id 34ca10faef96f3c660977612e46a8bf3755a53e9dd13820c0efc7cdabc6bf1c3 Nov 02 19:49:49.326967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:49.326937 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq"] Nov 02 19:49:49.374914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:49.374883 4966 generic.go:356] "Generic (PLEG): container finished" podID="92e01470-20f7-4ded-ad9b-63ec1848cf5d" containerID="904b5a6a54c1afa3a51f6cfc821f260e1687034947d17d2bb23826bcd5722b49" exitCode=0 Nov 02 19:49:49.375075 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:49.374971 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" event={"ID":"92e01470-20f7-4ded-ad9b-63ec1848cf5d","Type":"ContainerDied","Data":"904b5a6a54c1afa3a51f6cfc821f260e1687034947d17d2bb23826bcd5722b49"} Nov 02 19:49:49.376570 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:49.376542 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" event={"ID":"d0ab6e1b-42b4-4b21-9012-4dbd1be85462","Type":"ContainerStarted","Data":"34ca10faef96f3c660977612e46a8bf3755a53e9dd13820c0efc7cdabc6bf1c3"} Nov 02 19:49:50.388649 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.388615 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" event={"ID":"d0ab6e1b-42b4-4b21-9012-4dbd1be85462","Type":"ContainerStarted","Data":"f6ce41b77117689521165eaf6869162b01b1b78b39f1a5f9192968142a60412a"} Nov 02 19:49:50.503507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.498002 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" podStartSLOduration=2.497980752 podStartE2EDuration="2.497980752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:49:50.494556073 +0000 UTC m=+1296.443211912" watchObservedRunningTime="2026-11-02 19:49:50.497980752 +0000 UTC m=+1296.446636592" Nov 02 19:49:50.583811 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.583781 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:50.637533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.637501 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-xks-ca\") pod \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " Nov 02 19:49:50.637708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.637666 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e01470-20f7-4ded-ad9b-63ec1848cf5d-kube-api-access-mxmxl\" (UniqueName: \"kubernetes.io/projected/92e01470-20f7-4ded-ad9b-63ec1848cf5d-kube-api-access-mxmxl\") pod \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " Nov 02 19:49:50.637779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.637727 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-openshift-ca\") pod \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\" (UID: \"92e01470-20f7-4ded-ad9b-63ec1848cf5d\") " Nov 02 19:49:50.638074 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.638047 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-openshift-ca" pod "92e01470-20f7-4ded-ad9b-63ec1848cf5d" (UID: "92e01470-20f7-4ded-ad9b-63ec1848cf5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:50.638484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.638461 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:50.639794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.639730 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-xks-ca" pod "92e01470-20f7-4ded-ad9b-63ec1848cf5d" (UID: "92e01470-20f7-4ded-ad9b-63ec1848cf5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:50.639902 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.639878 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e01470-20f7-4ded-ad9b-63ec1848cf5d-kube-api-access-mxmxl" pod "92e01470-20f7-4ded-ad9b-63ec1848cf5d" (UID: "92e01470-20f7-4ded-ad9b-63ec1848cf5d"). InnerVolumeSpecName "kube-api-access-mxmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:50.740905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.740863 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e01470-20f7-4ded-ad9b-63ec1848cf5d-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:50.740905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:50.740901 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmxl\" (UniqueName: \"kubernetes.io/projected/92e01470-20f7-4ded-ad9b-63ec1848cf5d-kube-api-access-mxmxl\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:51.399731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:51.399694 4966 generic.go:356] "Generic (PLEG): container finished" podID="d0ab6e1b-42b4-4b21-9012-4dbd1be85462" containerID="f6ce41b77117689521165eaf6869162b01b1b78b39f1a5f9192968142a60412a" exitCode=0 Nov 02 19:49:51.400147 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:51.399777 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" event={"ID":"d0ab6e1b-42b4-4b21-9012-4dbd1be85462","Type":"ContainerDied","Data":"f6ce41b77117689521165eaf6869162b01b1b78b39f1a5f9192968142a60412a"} Nov 02 19:49:51.401475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:51.401454 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" event={"ID":"92e01470-20f7-4ded-ad9b-63ec1848cf5d","Type":"ContainerDied","Data":"a44de15711bc00d73a568d7380b9a2bf6e0f355697d1c97c9221497afacef593"} Nov 02 19:49:51.401601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:51.401478 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7" Nov 02 19:49:51.401601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:51.401490 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44de15711bc00d73a568d7380b9a2bf6e0f355697d1c97c9221497afacef593" Nov 02 19:49:52.616840 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.616814 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:52.670189 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.670158 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-openshift-ca\") pod \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " Nov 02 19:49:52.670368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.670194 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-xks-ca\") pod \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " Nov 02 19:49:52.670368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.670280 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-kube-api-access-q6n82\" (UniqueName: \"kubernetes.io/projected/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-kube-api-access-q6n82\") pod \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\" (UID: \"d0ab6e1b-42b4-4b21-9012-4dbd1be85462\") " Nov 02 19:49:52.670635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.670600 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-openshift-ca" pod "d0ab6e1b-42b4-4b21-9012-4dbd1be85462" (UID: "d0ab6e1b-42b4-4b21-9012-4dbd1be85462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:49:52.671160 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.671123 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:52.672664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.672596 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-xks-ca" pod "d0ab6e1b-42b4-4b21-9012-4dbd1be85462" (UID: "d0ab6e1b-42b4-4b21-9012-4dbd1be85462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:49:52.672664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.672637 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-kube-api-access-q6n82" pod "d0ab6e1b-42b4-4b21-9012-4dbd1be85462" (UID: "d0ab6e1b-42b4-4b21-9012-4dbd1be85462"). InnerVolumeSpecName "kube-api-access-q6n82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:49:52.772220 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.772184 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:52.772220 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:52.772219 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6n82\" (UniqueName: \"kubernetes.io/projected/d0ab6e1b-42b4-4b21-9012-4dbd1be85462-kube-api-access-q6n82\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:49:53.433112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:53.433082 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" Nov 02 19:49:53.433336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:53.433116 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq" event={"ID":"d0ab6e1b-42b4-4b21-9012-4dbd1be85462","Type":"ContainerDied","Data":"34ca10faef96f3c660977612e46a8bf3755a53e9dd13820c0efc7cdabc6bf1c3"} Nov 02 19:49:53.433336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:53.433163 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34ca10faef96f3c660977612e46a8bf3755a53e9dd13820c0efc7cdabc6bf1c3" Nov 02 19:49:56.114375 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.000775 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.114375 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.003598 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.268439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.265533 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 02 19:49:56.432384 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.424421 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.432384 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.424899 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 19:49:56.465517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.465469 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.487322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.487260 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.557324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.552372 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 19:49:56.557324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.552732 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.557324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.552981 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:56.671486 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:56.671424 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.131555 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.131443 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.132074 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.131838 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 19:49:57.135181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.132507 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:49:57.135181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.132571 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:49:57.376323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.360007 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.377600 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.377564 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 02 19:49:57.783311 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.604627 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": context deadline exceeded" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.783897 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.783971 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.688666 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.688695 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.714374 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.714432 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.795104 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.795166 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.795210 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.714410 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.714572 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.688083 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.728014 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.799609 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.714902 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.800723 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.800772 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:49:57.822352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.800808 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:49:57.883392 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:57.867313 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:49:58.154392 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:58.115600 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.154392 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:58.137408 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:58.154392 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:58.141391 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 19:49:58.375492 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:58.375445 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 19:49:58.477720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:58.477683 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:49:59.059322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.004254 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 19:49:59.059322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.005549 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 19:49:59.059322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.006088 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:59.059322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.038068 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 02 19:49:59.059322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.046825 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 02 19:49:59.069320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.063744 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 02 19:49:59.094320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.081948 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 19:49:59.147553 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.133558 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 19:49:59.277433 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.277401 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 19:49:59.278170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.147894 4966 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 02 19:49:59.300475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.148044 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:59.301468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.148069 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": context deadline exceeded" Nov 02 19:49:59.313334 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.147943 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 19:49:59.376322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.349832 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 19:49:59.376322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.374933 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 19:49:59.386282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.385224 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 19:49:59.388201 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.388164 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 19:49:59.404402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.404186 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 19:49:59.404402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.404249 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 19:49:59.413967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:49:59.413942 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 19:50:14.754286 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:50:14.754245 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:50:16.330933 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.330903 4966 scope.go:122] "RemoveContainer" containerID="410046de65dfce4f1b5cd2bfcb94e3e5e7070b005f4a73b0ebd0cf1a01067bd9" Nov 02 19:50:16.342841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.342819 4966 scope.go:122] "RemoveContainer" containerID="7ce7dc8b06985db06b0d56367dec551cb2b4c33dfe987ca850264e3bc607f84d" Nov 02 19:50:16.353606 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.353584 4966 scope.go:122] "RemoveContainer" containerID="8f2dd28785cc6023a180ef7c6f86692fe4b4039f6cd1601a060570bff0ee480d" Nov 02 19:50:16.364438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.364345 4966 scope.go:122] "RemoveContainer" containerID="e0ccc39be1e9433ff1c168cd1d831a48a538631f2a2dd8146b3f9a728b2a7094" Nov 02 19:50:16.378180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.378155 4966 scope.go:122] "RemoveContainer" containerID="bf2954401a34244d943d6b4e7aa633b1cbf7310b2e09ca148a04de2a4333a97a" Nov 02 19:50:16.391395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.391371 4966 scope.go:122] "RemoveContainer" containerID="d450354bae64e217f54b2ac9a188700dd38ebcd79a198852ef3430f463b92ac2" Nov 02 19:50:16.406263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.406164 4966 scope.go:122] "RemoveContainer" containerID="bb2441a95e4c30a71bcda836510dbd577e1c912303a10b0b27c0b85ac8e1cae4" Nov 02 19:50:16.419325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.419291 4966 scope.go:122] "RemoveContainer" containerID="3668f5a1bb104627fff0d6ba77abcfb834cde3320b71f7fc1d8d8c3e911e275d" Nov 02 19:50:16.432459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.432430 4966 scope.go:122] "RemoveContainer" containerID="1304b8bee89b2d906966e3cd249aeea464881b80e7c0c5781025f6661df7a70a" Nov 02 19:50:16.445728 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.445704 4966 scope.go:122] "RemoveContainer" containerID="2ed7e0cff74ec632c84cce65cbcfa053afb42df15e72cf849ea51c39ec2abcc0" Nov 02 19:50:16.455691 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.455673 4966 scope.go:122] "RemoveContainer" containerID="b1047fd3c52440ea0cff21b15e3590f83b46acf44bc1672563c71a92de2dbb99" Nov 02 19:50:16.466127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:16.466105 4966 scope.go:122] "RemoveContainer" containerID="545af9186588c80d02be993527f625b64ba9ed8c3d8a2a7ec3f39cb8dfa2f73b" Nov 02 19:50:19.943803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.943769 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n"] Nov 02 19:50:19.944762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.944732 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d0ab6e1b-42b4-4b21-9012-4dbd1be85462" containerName="revoke-keys" Nov 02 19:50:19.944762 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.944761 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ab6e1b-42b4-4b21-9012-4dbd1be85462" containerName="revoke-keys" Nov 02 19:50:19.944940 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.944783 4966 cpu_manager.go:414] "containerMap: removing container" podUID="92e01470-20f7-4ded-ad9b-63ec1848cf5d" containerName="revoke-keys" Nov 02 19:50:19.944940 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.944789 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e01470-20f7-4ded-ad9b-63ec1848cf5d" containerName="revoke-keys" Nov 02 19:50:19.945034 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.945015 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ab6e1b-42b4-4b21-9012-4dbd1be85462" containerName="revoke-keys" Nov 02 19:50:19.945070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.945042 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e01470-20f7-4ded-ad9b-63ec1848cf5d" containerName="revoke-keys" Nov 02 19:50:19.951939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:19.951919 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024161 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024130 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxtr\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-kube-api-access-kbxtr\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024227 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024257 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024292 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024453 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024492 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024615 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024785 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024768 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.024842 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.024799 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126206 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxtr\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-kube-api-access-kbxtr\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126352 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126381 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126442 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126524 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126552 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126647 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126830 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126801 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.126887 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.126830 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.127029 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.127004 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-credential-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.127287 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.127260 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-socket\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.127511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.127321 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-data\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.127594 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.127341 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istiod-ca-cert\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.127594 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.127386 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-certs\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.129415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.129388 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-envoy\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.130964 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.130940 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-podinfo\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.144576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.144546 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n"] Nov 02 19:50:20.180245 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.180213 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-token\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.180932 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.180898 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxtr\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-kube-api-access-kbxtr\") pod \"e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.267732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.267704 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:20.707159 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:50:20.707114 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f5dae8_66ca_497f_aec4_220ffc3ac4b8.slice/crio-1d25f6ae7218c0aa1157818039e13d72fc670a9a25102bcdb48c3a95eb3ee022 WatchSource:0}: Error finding container 1d25f6ae7218c0aa1157818039e13d72fc670a9a25102bcdb48c3a95eb3ee022: Status 404 returned error can't find the container with id 1d25f6ae7218c0aa1157818039e13d72fc670a9a25102bcdb48c3a95eb3ee022 Nov 02 19:50:20.708571 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.708547 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n"] Nov 02 19:50:20.709928 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.709894 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:50:20.710017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.709960 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:50:20.710017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:20.709990 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:50:21.636875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:21.636838 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" event={"ID":"59f5dae8-66ca-497f-aec4-220ffc3ac4b8","Type":"ContainerStarted","Data":"07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36"} Nov 02 19:50:21.637390 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:21.636905 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" event={"ID":"59f5dae8-66ca-497f-aec4-220ffc3ac4b8","Type":"ContainerStarted","Data":"1d25f6ae7218c0aa1157818039e13d72fc670a9a25102bcdb48c3a95eb3ee022"} Nov 02 19:50:21.802158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:21.802097 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podStartSLOduration=2.802078229 podStartE2EDuration="2.802078229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:50:21.79397756 +0000 UTC m=+1327.742633401" watchObservedRunningTime="2026-11-02 19:50:21.802078229 +0000 UTC m=+1327.750734068" Nov 02 19:50:22.268627 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:22.268586 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:22.270904 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:22.270876 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 19:50:22.271036 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:22.270949 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 19:50:23.272323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:23.272202 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 19:50:23.272323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:23.272273 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 19:50:24.271323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:24.270814 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 19:50:24.271323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:24.270917 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 19:50:25.268197 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:25.268156 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 19:50:25.268693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:25.268218 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 19:50:26.268988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:26.268945 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 19:50:26.269528 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:26.269037 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 19:50:27.268098 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:27.268056 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 19:50:27.268361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:27.268129 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 19:50:28.412662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:28.412634 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:28.413134 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:28.413123 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:28.413914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:50:28.413894 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 19:50:44.755630 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:50:44.755592 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:51:04.886699 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.886659 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx"] Nov 02 19:51:04.892707 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.892677 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.957624 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.957586 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqdvh\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-kube-api-access-sqdvh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.957814 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.957651 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.957881 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.957854 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22f389-14eb-45aa-b240-debdc9244ebf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.957923 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.957906 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.958117 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.958092 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.958171 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.958132 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.958259 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.958237 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.958350 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.958323 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:04.958404 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:04.958387 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.059660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059625 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdvh\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-kube-api-access-sqdvh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.059660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059667 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.059908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059727 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22f389-14eb-45aa-b240-debdc9244ebf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.059908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059748 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.059908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059841 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.059908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059884 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060129 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059932 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060129 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.059980 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060129 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.060020 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.060275 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060541 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.060518 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22f389-14eb-45aa-b240-debdc9244ebf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060679 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.060657 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060733 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.060678 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.060776 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.060743 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.062462 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.062437 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.062552 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.062455 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.098068 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.098031 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx"] Nov 02 19:51:05.127551 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.127509 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqdvh\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-kube-api-access-sqdvh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.127847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.127825 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b2ssx\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.213199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.213119 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:05.641206 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:51:05.641167 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a22f389_14eb_45aa_b240_debdc9244ebf.slice/crio-0fe372103b2b83deb861258279d797e1b566253ae9a85f5203df745129c13ee1 WatchSource:0}: Error finding container 0fe372103b2b83deb861258279d797e1b566253ae9a85f5203df745129c13ee1: Status 404 returned error can't find the container with id 0fe372103b2b83deb861258279d797e1b566253ae9a85f5203df745129c13ee1 Nov 02 19:51:05.643787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.643751 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx"] Nov 02 19:51:05.656310 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.656257 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:51:05.656435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.656383 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:51:05.656435 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.656430 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:51:05.705970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.705943 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2"] Nov 02 19:51:05.712141 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.712118 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.760528 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.760496 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:51:05.760724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.760623 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:51:05.763967 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.763935 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2"] Nov 02 19:51:05.770566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.770535 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.770732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.770599 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.770732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.770627 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssk5h\" (UniqueName: \"kubernetes.io/projected/d095a834-99db-415e-8fd8-873a3b8b33b6-kube-api-access-ssk5h\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.851536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.851501 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:51:05.851705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.851576 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:51:05.872723 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.872692 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.872723 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.872722 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssk5h\" (UniqueName: \"kubernetes.io/projected/d095a834-99db-415e-8fd8-873a3b8b33b6-kube-api-access-ssk5h\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.872968 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.872953 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.873382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.873360 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.875208 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.875189 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:05.925272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:05.925195 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssk5h\" (UniqueName: \"kubernetes.io/projected/d095a834-99db-415e-8fd8-873a3b8b33b6-kube-api-access-ssk5h\") pod \"maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:06.041254 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:06.041231 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:06.235841 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:06.235805 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" event={"ID":"1a22f389-14eb-45aa-b240-debdc9244ebf","Type":"ContainerStarted","Data":"44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6"} Nov 02 19:51:06.236013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:06.235863 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" event={"ID":"1a22f389-14eb-45aa-b240-debdc9244ebf","Type":"ContainerStarted","Data":"0fe372103b2b83deb861258279d797e1b566253ae9a85f5203df745129c13ee1"} Nov 02 19:51:06.356642 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:06.353460 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podStartSLOduration=2.353436922 podStartE2EDuration="2.353436922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:06.345852019 +0000 UTC m=+1372.294507859" watchObservedRunningTime="2026-11-02 19:51:06.353436922 +0000 UTC m=+1372.302092762" Nov 02 19:51:06.469401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:06.469344 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2"] Nov 02 19:51:06.470681 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:51:06.470652 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd095a834_99db_415e_8fd8_873a3b8b33b6.slice/crio-182a0e5325aa391f2d8a8cce6ef0deb1b8fa65577f67f2ab40c1d9780f38eaca WatchSource:0}: Error finding container 182a0e5325aa391f2d8a8cce6ef0deb1b8fa65577f67f2ab40c1d9780f38eaca: Status 404 returned error can't find the container with id 182a0e5325aa391f2d8a8cce6ef0deb1b8fa65577f67f2ab40c1d9780f38eaca Nov 02 19:51:07.213395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:07.213309 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:07.214543 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:07.214508 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 19:51:07.214674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:07.214565 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 19:51:07.289320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:07.288367 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" event={"ID":"d095a834-99db-415e-8fd8-873a3b8b33b6","Type":"ContainerStarted","Data":"19594b231c3d78655cede0a7c83bdacbb5f40b3f701fd85fb11ca7d3f8116b95"} Nov 02 19:51:07.289320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:07.288430 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" event={"ID":"d095a834-99db-415e-8fd8-873a3b8b33b6","Type":"ContainerStarted","Data":"182a0e5325aa391f2d8a8cce6ef0deb1b8fa65577f67f2ab40c1d9780f38eaca"} Nov 02 19:51:08.213905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:08.213865 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 19:51:08.214433 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:08.213922 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 19:51:08.300228 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:08.300185 4966 generic.go:356] "Generic (PLEG): container finished" podID="d095a834-99db-415e-8fd8-873a3b8b33b6" containerID="19594b231c3d78655cede0a7c83bdacbb5f40b3f701fd85fb11ca7d3f8116b95" exitCode=0 Nov 02 19:51:08.301281 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:08.301258 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" event={"ID":"d095a834-99db-415e-8fd8-873a3b8b33b6","Type":"ContainerDied","Data":"19594b231c3d78655cede0a7c83bdacbb5f40b3f701fd85fb11ca7d3f8116b95"} Nov 02 19:51:09.214262 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.214225 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 19:51:09.214671 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.214310 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 19:51:09.565456 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.565426 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:09.661321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.660914 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d095a834-99db-415e-8fd8-873a3b8b33b6-kube-api-access-ssk5h\" (UniqueName: \"kubernetes.io/projected/d095a834-99db-415e-8fd8-873a3b8b33b6-kube-api-access-ssk5h\") pod \"d095a834-99db-415e-8fd8-873a3b8b33b6\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " Nov 02 19:51:09.661321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.661012 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-xks-ca\") pod \"d095a834-99db-415e-8fd8-873a3b8b33b6\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " Nov 02 19:51:09.661321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.661055 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-openshift-ca\") pod \"d095a834-99db-415e-8fd8-873a3b8b33b6\" (UID: \"d095a834-99db-415e-8fd8-873a3b8b33b6\") " Nov 02 19:51:09.662580 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.662530 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-openshift-ca" pod "d095a834-99db-415e-8fd8-873a3b8b33b6" (UID: "d095a834-99db-415e-8fd8-873a3b8b33b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:09.664200 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.664175 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d095a834-99db-415e-8fd8-873a3b8b33b6-kube-api-access-ssk5h" pod "d095a834-99db-415e-8fd8-873a3b8b33b6" (UID: "d095a834-99db-415e-8fd8-873a3b8b33b6"). InnerVolumeSpecName "kube-api-access-ssk5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:09.664698 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.664671 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-xks-ca" pod "d095a834-99db-415e-8fd8-873a3b8b33b6" (UID: "d095a834-99db-415e-8fd8-873a3b8b33b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:09.763493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.763405 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssk5h\" (UniqueName: \"kubernetes.io/projected/d095a834-99db-415e-8fd8-873a3b8b33b6-kube-api-access-ssk5h\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:09.763493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.763439 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:09.763493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:09.763450 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d095a834-99db-415e-8fd8-873a3b8b33b6-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:10.213701 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:10.213665 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 19:51:10.214016 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:10.213758 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 19:51:10.322989 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:10.322961 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" Nov 02 19:51:10.323439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:10.322998 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2" event={"ID":"d095a834-99db-415e-8fd8-873a3b8b33b6","Type":"ContainerDied","Data":"182a0e5325aa391f2d8a8cce6ef0deb1b8fa65577f67f2ab40c1d9780f38eaca"} Nov 02 19:51:10.323439 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:10.323043 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="182a0e5325aa391f2d8a8cce6ef0deb1b8fa65577f67f2ab40c1d9780f38eaca" Nov 02 19:51:11.213521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:11.213479 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 19:51:11.213781 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:11.213540 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 19:51:12.214542 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:12.214504 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 19:51:12.214938 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:12.214572 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 19:51:13.214291 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:13.214251 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 19:51:13.214478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:13.214339 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 19:51:14.377195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:14.377143 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:14.378482 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:14.377742 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:14.379754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:14.379731 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 19:51:14.756263 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:51:14.756231 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:51:16.591398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:16.591366 4966 scope.go:122] "RemoveContainer" containerID="d0e11db5fa00faa2cfc16f37dbee72eb02b14809415ca3a141d78ea6f82e1ae1" Nov 02 19:51:18.292323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:18.292273 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng"] Nov 02 19:51:18.421399 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:18.421368 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh"] Nov 02 19:51:22.188415 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.188383 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp"] Nov 02 19:51:22.189067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.189051 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d095a834-99db-415e-8fd8-873a3b8b33b6" containerName="revoke-keys" Nov 02 19:51:22.189111 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.189068 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d095a834-99db-415e-8fd8-873a3b8b33b6" containerName="revoke-keys" Nov 02 19:51:22.189225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.189215 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d095a834-99db-415e-8fd8-873a3b8b33b6" containerName="revoke-keys" Nov 02 19:51:22.195188 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.195170 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.239931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.239899 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp"] Nov 02 19:51:22.279485 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.279450 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:51:22.279716 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.279457 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:51:22.289521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.289485 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:51:22.289521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.289486 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:51:22.346356 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.346322 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.346629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.346606 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4l8\" (UniqueName: \"kubernetes.io/projected/e3b44220-5e73-466c-91e7-d6f5d70eb400-kube-api-access-vs4l8\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.346835 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.346819 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.448010 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.447908 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.448198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.448069 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.448198 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.448119 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4l8\" (UniqueName: \"kubernetes.io/projected/e3b44220-5e73-466c-91e7-d6f5d70eb400-kube-api-access-vs4l8\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.448928 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.448899 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.450332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.450272 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.500789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.500754 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4l8\" (UniqueName: \"kubernetes.io/projected/e3b44220-5e73-466c-91e7-d6f5d70eb400-kube-api-access-vs4l8\") pod \"maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.510581 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.510552 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:22.890975 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:51:22.890935 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b44220_5e73_466c_91e7_d6f5d70eb400.slice/crio-7f6cc12c08dbd46bae3ab16d530d5d32c0662f8393a7c5530a3eaa7b72e3b3f0 WatchSource:0}: Error finding container 7f6cc12c08dbd46bae3ab16d530d5d32c0662f8393a7c5530a3eaa7b72e3b3f0: Status 404 returned error can't find the container with id 7f6cc12c08dbd46bae3ab16d530d5d32c0662f8393a7c5530a3eaa7b72e3b3f0 Nov 02 19:51:22.893405 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:22.893378 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp"] Nov 02 19:51:23.293615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.293557 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" podUID="9c763680-0ea9-480e-8b0b-c6fe9b800529" containerName="payload-processing" containerID="cri-o://336ba983273504b3fedbf6d4fb1961a166515149a3d74e9305e9489cf3b7fc7e" gracePeriod=25 Nov 02 19:51:23.358380 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.358339 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z"] Nov 02 19:51:23.363125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.363096 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.415946 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.415923 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z"] Nov 02 19:51:23.422114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.422075 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" podUID="c0ee65ed-8d42-43a1-837e-72f14e451d43" containerName="payload-pre-processing" containerID="cri-o://753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020" gracePeriod=25 Nov 02 19:51:23.467351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.465444 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.467351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.465580 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htdc\" (UniqueName: \"kubernetes.io/projected/ec9a8658-d438-452c-9b2b-091f65053b91-kube-api-access-6htdc\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.467351 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.465704 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.501420 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.501389 4966 generic.go:356] "Generic (PLEG): container finished" podID="9c763680-0ea9-480e-8b0b-c6fe9b800529" containerID="336ba983273504b3fedbf6d4fb1961a166515149a3d74e9305e9489cf3b7fc7e" exitCode=0 Nov 02 19:51:23.501566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.501461 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" event={"ID":"9c763680-0ea9-480e-8b0b-c6fe9b800529","Type":"ContainerDied","Data":"336ba983273504b3fedbf6d4fb1961a166515149a3d74e9305e9489cf3b7fc7e"} Nov 02 19:51:23.503239 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.503215 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" event={"ID":"e3b44220-5e73-466c-91e7-d6f5d70eb400","Type":"ContainerStarted","Data":"a4ab87284a3b562fdb55f483cab5ac6c853e5322d8fbac165a00a55a19e36a5a"} Nov 02 19:51:23.503396 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.503252 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" event={"ID":"e3b44220-5e73-466c-91e7-d6f5d70eb400","Type":"ContainerStarted","Data":"7f6cc12c08dbd46bae3ab16d530d5d32c0662f8393a7c5530a3eaa7b72e3b3f0"} Nov 02 19:51:23.568381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.567469 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.568381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.567705 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.568381 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.567809 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htdc\" (UniqueName: \"kubernetes.io/projected/ec9a8658-d438-452c-9b2b-091f65053b91-kube-api-access-6htdc\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.569738 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.569276 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.594052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.594022 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.614789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.614720 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" podStartSLOduration=1.6146985969999998 podStartE2EDuration="1.614698597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:23.608289009 +0000 UTC m=+1389.556944848" watchObservedRunningTime="2026-11-02 19:51:23.614698597 +0000 UTC m=+1389.563354450" Nov 02 19:51:23.625061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.624977 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htdc\" (UniqueName: \"kubernetes.io/projected/ec9a8658-d438-452c-9b2b-091f65053b91-kube-api-access-6htdc\") pod \"maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.663955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.663890 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:51:23.684741 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.684708 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:23.769635 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.769606 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:51:23.770536 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.770509 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c763680-0ea9-480e-8b0b-c6fe9b800529-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c763680-0ea9-480e-8b0b-c6fe9b800529-plugins-config-volume\") pod \"9c763680-0ea9-480e-8b0b-c6fe9b800529\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " Nov 02 19:51:23.770666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.770565 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c763680-0ea9-480e-8b0b-c6fe9b800529-kube-api-access-klhvg\" (UniqueName: \"kubernetes.io/projected/9c763680-0ea9-480e-8b0b-c6fe9b800529-kube-api-access-klhvg\") pod \"9c763680-0ea9-480e-8b0b-c6fe9b800529\" (UID: \"9c763680-0ea9-480e-8b0b-c6fe9b800529\") " Nov 02 19:51:23.773919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.773622 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c763680-0ea9-480e-8b0b-c6fe9b800529-kube-api-access-klhvg" pod "9c763680-0ea9-480e-8b0b-c6fe9b800529" (UID: "9c763680-0ea9-480e-8b0b-c6fe9b800529"). InnerVolumeSpecName "kube-api-access-klhvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:23.774240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.774200 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c763680-0ea9-480e-8b0b-c6fe9b800529-plugins-config-volume" pod "9c763680-0ea9-480e-8b0b-c6fe9b800529" (UID: "9c763680-0ea9-480e-8b0b-c6fe9b800529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:23.872711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.871545 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ee65ed-8d42-43a1-837e-72f14e451d43-kube-api-access-8skx7\" (UniqueName: \"kubernetes.io/projected/c0ee65ed-8d42-43a1-837e-72f14e451d43-kube-api-access-8skx7\") pod \"c0ee65ed-8d42-43a1-837e-72f14e451d43\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " Nov 02 19:51:23.872711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.871604 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ee65ed-8d42-43a1-837e-72f14e451d43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee65ed-8d42-43a1-837e-72f14e451d43-plugins-config-volume\") pod \"c0ee65ed-8d42-43a1-837e-72f14e451d43\" (UID: \"c0ee65ed-8d42-43a1-837e-72f14e451d43\") " Nov 02 19:51:23.872711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.872087 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee65ed-8d42-43a1-837e-72f14e451d43-plugins-config-volume" pod "c0ee65ed-8d42-43a1-837e-72f14e451d43" (UID: "c0ee65ed-8d42-43a1-837e-72f14e451d43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:23.872711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.872637 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee65ed-8d42-43a1-837e-72f14e451d43-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:23.872711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.872664 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c763680-0ea9-480e-8b0b-c6fe9b800529-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:23.872711 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.872682 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhvg\" (UniqueName: \"kubernetes.io/projected/9c763680-0ea9-480e-8b0b-c6fe9b800529-kube-api-access-klhvg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:23.874050 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.874023 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ee65ed-8d42-43a1-837e-72f14e451d43-kube-api-access-8skx7" pod "c0ee65ed-8d42-43a1-837e-72f14e451d43" (UID: "c0ee65ed-8d42-43a1-837e-72f14e451d43"). InnerVolumeSpecName "kube-api-access-8skx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:23.973987 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:23.973957 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skx7\" (UniqueName: \"kubernetes.io/projected/c0ee65ed-8d42-43a1-837e-72f14e451d43-kube-api-access-8skx7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:24.086772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.086491 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z"] Nov 02 19:51:24.514813 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.514732 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" event={"ID":"9c763680-0ea9-480e-8b0b-c6fe9b800529","Type":"ContainerDied","Data":"35f19a1afcca14e956f96c107042ac04df39685fd39a990bd92aa48d1e719745"} Nov 02 19:51:24.514813 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.514799 4966 scope.go:122] "RemoveContainer" containerID="336ba983273504b3fedbf6d4fb1961a166515149a3d74e9305e9489cf3b7fc7e" Nov 02 19:51:24.515340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.514739 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng" Nov 02 19:51:24.516598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.516571 4966 generic.go:356] "Generic (PLEG): container finished" podID="c0ee65ed-8d42-43a1-837e-72f14e451d43" containerID="753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020" exitCode=0 Nov 02 19:51:24.516736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.516631 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" event={"ID":"c0ee65ed-8d42-43a1-837e-72f14e451d43","Type":"ContainerDied","Data":"753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020"} Nov 02 19:51:24.516736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.516652 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" Nov 02 19:51:24.516736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.516692 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh" event={"ID":"c0ee65ed-8d42-43a1-837e-72f14e451d43","Type":"ContainerDied","Data":"bbeeecdb3cf7baabd3980e07800b4a2670f6a075b63ab23786b529d1026ea6b4"} Nov 02 19:51:24.519076 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.519046 4966 generic.go:356] "Generic (PLEG): container finished" podID="e3b44220-5e73-466c-91e7-d6f5d70eb400" containerID="a4ab87284a3b562fdb55f483cab5ac6c853e5322d8fbac165a00a55a19e36a5a" exitCode=0 Nov 02 19:51:24.519246 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.519187 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" event={"ID":"e3b44220-5e73-466c-91e7-d6f5d70eb400","Type":"ContainerDied","Data":"a4ab87284a3b562fdb55f483cab5ac6c853e5322d8fbac165a00a55a19e36a5a"} Nov 02 19:51:24.521344 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.521319 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" event={"ID":"ec9a8658-d438-452c-9b2b-091f65053b91","Type":"ContainerStarted","Data":"3112dcb98464d46c5e15f443aa24594faa76c84c1524108588538954e34dae2e"} Nov 02 19:51:24.521450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.521364 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" event={"ID":"ec9a8658-d438-452c-9b2b-091f65053b91","Type":"ContainerStarted","Data":"c57c834b93b6b2648220df28e1d7a334aa459422e13df44944308406931ac26e"} Nov 02 19:51:24.527688 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.527669 4966 scope.go:122] "RemoveContainer" containerID="753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020" Nov 02 19:51:24.538965 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.538948 4966 scope.go:122] "RemoveContainer" containerID="753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020" Nov 02 19:51:24.539278 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:51:24.539258 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020\": container with ID starting with 753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020 not found: ID does not exist" containerID="753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020" Nov 02 19:51:24.539382 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.539291 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020"} err="failed to get container status \"753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020\": rpc error: code = NotFound desc = could not find container \"753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020\": container with ID starting with 753c08a9b70cf4d67c24866ff7784a65dc7f4885f97117d1517d030c1bdfa020 not found: ID does not exist" Nov 02 19:51:24.777579 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.777544 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh"] Nov 02 19:51:24.783190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.783160 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d23a1e27-dcdd858c5-94nfh"] Nov 02 19:51:24.878003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.877943 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" podStartSLOduration=1.87793017 podStartE2EDuration="1.87793017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:24.875052892 +0000 UTC m=+1390.823708732" watchObservedRunningTime="2026-11-02 19:51:24.87793017 +0000 UTC m=+1390.826586012" Nov 02 19:51:24.982955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.982908 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng"] Nov 02 19:51:24.989475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:24.989435 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d23a1e27-f4c86f7d9-vzhng"] Nov 02 19:51:25.535311 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.535263 4966 generic.go:356] "Generic (PLEG): container finished" podID="ec9a8658-d438-452c-9b2b-091f65053b91" containerID="3112dcb98464d46c5e15f443aa24594faa76c84c1524108588538954e34dae2e" exitCode=0 Nov 02 19:51:25.535783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.535328 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" event={"ID":"ec9a8658-d438-452c-9b2b-091f65053b91","Type":"ContainerDied","Data":"3112dcb98464d46c5e15f443aa24594faa76c84c1524108588538954e34dae2e"} Nov 02 19:51:25.754242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.754209 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:25.813621 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.813533 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-openshift-ca\") pod \"e3b44220-5e73-466c-91e7-d6f5d70eb400\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " Nov 02 19:51:25.813812 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.813654 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-xks-ca\") pod \"e3b44220-5e73-466c-91e7-d6f5d70eb400\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " Nov 02 19:51:25.813812 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.813699 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b44220-5e73-466c-91e7-d6f5d70eb400-kube-api-access-vs4l8\" (UniqueName: \"kubernetes.io/projected/e3b44220-5e73-466c-91e7-d6f5d70eb400-kube-api-access-vs4l8\") pod \"e3b44220-5e73-466c-91e7-d6f5d70eb400\" (UID: \"e3b44220-5e73-466c-91e7-d6f5d70eb400\") " Nov 02 19:51:25.813957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.813918 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-openshift-ca" pod "e3b44220-5e73-466c-91e7-d6f5d70eb400" (UID: "e3b44220-5e73-466c-91e7-d6f5d70eb400"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:25.814379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.814358 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:25.815919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.815896 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-xks-ca" pod "e3b44220-5e73-466c-91e7-d6f5d70eb400" (UID: "e3b44220-5e73-466c-91e7-d6f5d70eb400"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:25.816036 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.815996 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b44220-5e73-466c-91e7-d6f5d70eb400-kube-api-access-vs4l8" pod "e3b44220-5e73-466c-91e7-d6f5d70eb400" (UID: "e3b44220-5e73-466c-91e7-d6f5d70eb400"). InnerVolumeSpecName "kube-api-access-vs4l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:25.915652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.915607 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b44220-5e73-466c-91e7-d6f5d70eb400-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:25.915652 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:25.915652 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4l8\" (UniqueName: \"kubernetes.io/projected/e3b44220-5e73-466c-91e7-d6f5d70eb400-kube-api-access-vs4l8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:26.050052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.050012 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2"] Nov 02 19:51:26.055530 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.055501 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-effce9fb-rjlz2"] Nov 02 19:51:26.186763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.186682 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c763680-0ea9-480e-8b0b-c6fe9b800529" path="/var/lib/kubelet/pods/9c763680-0ea9-480e-8b0b-c6fe9b800529/volumes" Nov 02 19:51:26.187326 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.187279 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4865ee-8920-4465-9cb6-341174df580d" path="/var/lib/kubelet/pods/aa4865ee-8920-4465-9cb6-341174df580d/volumes" Nov 02 19:51:26.187849 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.187826 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ee65ed-8d42-43a1-837e-72f14e451d43" path="/var/lib/kubelet/pods/c0ee65ed-8d42-43a1-837e-72f14e451d43/volumes" Nov 02 19:51:26.548729 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.548695 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" Nov 02 19:51:26.549182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.548699 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp" event={"ID":"e3b44220-5e73-466c-91e7-d6f5d70eb400","Type":"ContainerDied","Data":"7f6cc12c08dbd46bae3ab16d530d5d32c0662f8393a7c5530a3eaa7b72e3b3f0"} Nov 02 19:51:26.549182 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.548853 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f6cc12c08dbd46bae3ab16d530d5d32c0662f8393a7c5530a3eaa7b72e3b3f0" Nov 02 19:51:26.720863 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.720839 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:26.728709 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.728686 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-xks-ca\") pod \"ec9a8658-d438-452c-9b2b-091f65053b91\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " Nov 02 19:51:26.728820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.728788 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-openshift-ca\") pod \"ec9a8658-d438-452c-9b2b-091f65053b91\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " Nov 02 19:51:26.728901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.728887 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9a8658-d438-452c-9b2b-091f65053b91-kube-api-access-6htdc\" (UniqueName: \"kubernetes.io/projected/ec9a8658-d438-452c-9b2b-091f65053b91-kube-api-access-6htdc\") pod \"ec9a8658-d438-452c-9b2b-091f65053b91\" (UID: \"ec9a8658-d438-452c-9b2b-091f65053b91\") " Nov 02 19:51:26.729145 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.729120 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-openshift-ca" pod "ec9a8658-d438-452c-9b2b-091f65053b91" (UID: "ec9a8658-d438-452c-9b2b-091f65053b91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:26.729487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.729468 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:26.730685 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.730666 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-xks-ca" pod "ec9a8658-d438-452c-9b2b-091f65053b91" (UID: "ec9a8658-d438-452c-9b2b-091f65053b91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:26.730924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.730906 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9a8658-d438-452c-9b2b-091f65053b91-kube-api-access-6htdc" pod "ec9a8658-d438-452c-9b2b-091f65053b91" (UID: "ec9a8658-d438-452c-9b2b-091f65053b91"). InnerVolumeSpecName "kube-api-access-6htdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:26.831173 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.831086 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9a8658-d438-452c-9b2b-091f65053b91-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:26.831173 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:26.831116 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6htdc\" (UniqueName: \"kubernetes.io/projected/ec9a8658-d438-452c-9b2b-091f65053b91-kube-api-access-6htdc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:27.562511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:27.562474 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" event={"ID":"ec9a8658-d438-452c-9b2b-091f65053b91","Type":"ContainerDied","Data":"c57c834b93b6b2648220df28e1d7a334aa459422e13df44944308406931ac26e"} Nov 02 19:51:27.563014 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:27.562526 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57c834b93b6b2648220df28e1d7a334aa459422e13df44944308406931ac26e" Nov 02 19:51:27.563014 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:27.562486 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z" Nov 02 19:51:30.099026 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.098988 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66"] Nov 02 19:51:30.101225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.101195 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e3b44220-5e73-466c-91e7-d6f5d70eb400" containerName="revoke-keys" Nov 02 19:51:30.101418 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.101401 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b44220-5e73-466c-91e7-d6f5d70eb400" containerName="revoke-keys" Nov 02 19:51:30.101548 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.101534 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c0ee65ed-8d42-43a1-837e-72f14e451d43" containerName="payload-pre-processing" Nov 02 19:51:30.101637 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.101622 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ee65ed-8d42-43a1-837e-72f14e451d43" containerName="payload-pre-processing" Nov 02 19:51:30.101850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.101837 4966 cpu_manager.go:414] "containerMap: removing container" podUID="9c763680-0ea9-480e-8b0b-c6fe9b800529" containerName="payload-processing" Nov 02 19:51:30.101948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.101937 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c763680-0ea9-480e-8b0b-c6fe9b800529" containerName="payload-processing" Nov 02 19:51:30.102079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.102069 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ec9a8658-d438-452c-9b2b-091f65053b91" containerName="revoke-keys" Nov 02 19:51:30.102153 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.102143 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9a8658-d438-452c-9b2b-091f65053b91" containerName="revoke-keys" Nov 02 19:51:30.102600 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.102582 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c763680-0ea9-480e-8b0b-c6fe9b800529" containerName="payload-processing" Nov 02 19:51:30.102726 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.102715 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9a8658-d438-452c-9b2b-091f65053b91" containerName="revoke-keys" Nov 02 19:51:30.102817 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.102807 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b44220-5e73-466c-91e7-d6f5d70eb400" containerName="revoke-keys" Nov 02 19:51:30.102892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.102882 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ee65ed-8d42-43a1-837e-72f14e451d43" containerName="payload-pre-processing" Nov 02 19:51:30.120090 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.120051 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.145321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.139794 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66"] Nov 02 19:51:30.174666 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.174631 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:51:30.198169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.196583 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:51:30.232825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.232793 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:51:30.233902 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.233877 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:51:30.250052 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.250012 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.364366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.359912 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvzr\" (UniqueName: \"kubernetes.io/projected/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-kube-api-access-bqvzr\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.364366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.360159 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.462383 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.462344 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvzr\" (UniqueName: \"kubernetes.io/projected/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-kube-api-access-bqvzr\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.463103 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.463076 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.463835 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.463817 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.464928 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.464824 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.480937 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.480864 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.515996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.515945 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvzr\" (UniqueName: \"kubernetes.io/projected/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-kube-api-access-bqvzr\") pod \"maas-api-revoke-sub-22cb1d10-rdp66\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:30.726039 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:30.725995 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:31.111276 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:51:31.111242 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef0a458_1093_4de3_8c1c_8ab62d0fd8ae.slice/crio-d55378cbe31bf67284ffd570fadb4d3c28bdf09fa7aa2c829320c57268372fa3 WatchSource:0}: Error finding container d55378cbe31bf67284ffd570fadb4d3c28bdf09fa7aa2c829320c57268372fa3: Status 404 returned error can't find the container with id d55378cbe31bf67284ffd570fadb4d3c28bdf09fa7aa2c829320c57268372fa3 Nov 02 19:51:31.111700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:31.111628 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66"] Nov 02 19:51:31.605054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:31.605014 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" event={"ID":"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae","Type":"ContainerStarted","Data":"e37f78aa5be137dbaa34ae0203b7889c977ff6c8b96c17f6644cf4f23fa1aa63"} Nov 02 19:51:31.605241 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:31.605077 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" event={"ID":"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae","Type":"ContainerStarted","Data":"d55378cbe31bf67284ffd570fadb4d3c28bdf09fa7aa2c829320c57268372fa3"} Nov 02 19:51:31.712458 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:31.712343 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" podStartSLOduration=1.712291542 podStartE2EDuration="1.712291542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:31.709667086 +0000 UTC m=+1397.658322941" watchObservedRunningTime="2026-11-02 19:51:31.712291542 +0000 UTC m=+1397.660947380" Nov 02 19:51:32.617395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:32.617350 4966 generic.go:356] "Generic (PLEG): container finished" podID="2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" containerID="e37f78aa5be137dbaa34ae0203b7889c977ff6c8b96c17f6644cf4f23fa1aa63" exitCode=0 Nov 02 19:51:32.617834 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:32.617432 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" event={"ID":"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae","Type":"ContainerDied","Data":"e37f78aa5be137dbaa34ae0203b7889c977ff6c8b96c17f6644cf4f23fa1aa63"} Nov 02 19:51:33.802599 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.802570 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:33.920172 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.920082 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-openshift-ca\") pod \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " Nov 02 19:51:33.920631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.920603 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-openshift-ca" pod "2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" (UID: "2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:51:33.921010 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.920986 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-xks-ca\") pod \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " Nov 02 19:51:33.921130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.921028 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-kube-api-access-bqvzr\" (UniqueName: \"kubernetes.io/projected/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-kube-api-access-bqvzr\") pod \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\" (UID: \"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae\") " Nov 02 19:51:33.921717 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.921666 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:33.924343 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.924293 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-xks-ca" pod "2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" (UID: "2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:51:33.924612 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:33.924471 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-kube-api-access-bqvzr" pod "2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" (UID: "2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae"). InnerVolumeSpecName "kube-api-access-bqvzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:51:34.023559 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:34.023527 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:34.023559 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:34.023554 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqvzr\" (UniqueName: \"kubernetes.io/projected/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae-kube-api-access-bqvzr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:51:34.360402 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:51:34.360367 4966 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef0a458_1093_4de3_8c1c_8ab62d0fd8ae.slice\": RecentStats: unable to find data in memory cache]" Nov 02 19:51:34.640361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:34.640233 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" Nov 02 19:51:34.640361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:34.640270 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66" event={"ID":"2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae","Type":"ContainerDied","Data":"d55378cbe31bf67284ffd570fadb4d3c28bdf09fa7aa2c829320c57268372fa3"} Nov 02 19:51:34.640361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:34.640342 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d55378cbe31bf67284ffd570fadb4d3c28bdf09fa7aa2c829320c57268372fa3" Nov 02 19:51:44.759017 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:51:44.758982 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:51:50.274027 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:50.273992 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2"] Nov 02 19:51:50.320082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:50.320054 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d23a1e27-7tmr2"] Nov 02 19:51:52.184969 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:52.184934 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d095a834-99db-415e-8fd8-873a3b8b33b6" path="/var/lib/kubelet/pods/d095a834-99db-415e-8fd8-873a3b8b33b6/volumes" Nov 02 19:51:53.088022 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.087982 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7"] Nov 02 19:51:53.089050 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.089025 4966 cpu_manager.go:414] "containerMap: removing container" podUID="2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" containerName="revoke-keys" Nov 02 19:51:53.089154 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.089054 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" containerName="revoke-keys" Nov 02 19:51:53.089279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.089266 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" containerName="revoke-keys" Nov 02 19:51:53.096084 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.096060 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.146376 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.146330 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d368a053-openshift-default-dockercfg-4spbc\"" Nov 02 19:51:53.202546 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.202506 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7"] Nov 02 19:51:53.266994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.266926 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-credential-socket\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.271500 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-podinfo\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.271625 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3fa1b-d973-4e78-8340-a204313c171f-istiod-ca-cert\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.271681 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-envoy\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.271745 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-data\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.271815 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-socket\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.271957 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-certs\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.272035 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5gv4\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-kube-api-access-s5gv4\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.277195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.272127 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-token\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373437 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373328 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-certs\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373437 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373402 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5gv4\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-kube-api-access-s5gv4\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373475 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-token\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373512 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-credential-socket\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373552 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-podinfo\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373648 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3fa1b-d973-4e78-8340-a204313c171f-istiod-ca-cert\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373703 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-envoy\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373765 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-data\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.373931 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373836 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-socket\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.374095 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.373959 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-credential-socket\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.374598 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.374570 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3fa1b-d973-4e78-8340-a204313c171f-istiod-ca-cert\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.374803 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.374782 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-data\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.374874 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.374813 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-socket\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.374874 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.374824 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-certs\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.376889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.376863 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-podinfo\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.381597 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.381568 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-envoy\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.445077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.445042 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5gv4\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-kube-api-access-s5gv4\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.445535 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.445511 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-token\") pod \"e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:53.715559 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:53.715474 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:54.161012 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:51:54.160979 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb3fa1b_d973_4e78_8340_a204313c171f.slice/crio-f23872dc62b8efc224e14b781a13cad3e2293ffea2c01189d6d974dc40a430c6 WatchSource:0}: Error finding container f23872dc62b8efc224e14b781a13cad3e2293ffea2c01189d6d974dc40a430c6: Status 404 returned error can't find the container with id f23872dc62b8efc224e14b781a13cad3e2293ffea2c01189d6d974dc40a430c6 Nov 02 19:51:54.161941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:54.161911 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7"] Nov 02 19:51:54.164417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:54.164386 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:51:54.164521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:54.164456 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:51:54.164521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:54.164484 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:51:54.853142 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:54.853106 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" event={"ID":"5eb3fa1b-d973-4e78-8340-a204313c171f","Type":"ContainerStarted","Data":"c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb"} Nov 02 19:51:54.853708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:54.853199 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" event={"ID":"5eb3fa1b-d973-4e78-8340-a204313c171f","Type":"ContainerStarted","Data":"f23872dc62b8efc224e14b781a13cad3e2293ffea2c01189d6d974dc40a430c6"} Nov 02 19:51:55.007835 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:55.007771 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" podStartSLOduration=2.007753114 podStartE2EDuration="2.007753114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:55.00691016 +0000 UTC m=+1420.955565998" watchObservedRunningTime="2026-11-02 19:51:55.007753114 +0000 UTC m=+1420.956408952" Nov 02 19:51:55.715865 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:55.715830 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:55.721388 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:55.721362 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:55.862349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:55.862290 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:51:55.863163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:51:55.863142 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:52:00.927327 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:00.927250 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp"] Nov 02 19:52:00.937844 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:00.937353 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8ffab69-9sbbp"] Nov 02 19:52:01.943750 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:01.943717 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z"] Nov 02 19:52:01.992446 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:01.992407 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6d4e9ef2-4fv2z"] Nov 02 19:52:02.190953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:02.190913 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b44220-5e73-466c-91e7-d6f5d70eb400" path="/var/lib/kubelet/pods/e3b44220-5e73-466c-91e7-d6f5d70eb400/volumes" Nov 02 19:52:02.191776 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:02.191751 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9a8658-d438-452c-9b2b-091f65053b91" path="/var/lib/kubelet/pods/ec9a8658-d438-452c-9b2b-091f65053b91/volumes" Nov 02 19:52:06.049017 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:06.048981 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w"] Nov 02 19:52:06.097828 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:06.097781 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4512810e-nng9w"] Nov 02 19:52:06.188791 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:06.188755 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c04566-fcff-481f-945f-f2329c8d3897" path="/var/lib/kubelet/pods/a9c04566-fcff-481f-945f-f2329c8d3897/volumes" Nov 02 19:52:11.920395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:11.920345 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg"] Nov 02 19:52:11.929898 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:11.929869 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:11.984734 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:11.984703 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-46f86d84-dockercfg-cdcbm\"" Nov 02 19:52:12.008748 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.008708 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfr9p\" (UniqueName: \"kubernetes.io/projected/60192f85-e356-4ca2-afcb-572c3e47407d-kube-api-access-rfr9p\") pod \"payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:12.009403 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.009382 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60192f85-e356-4ca2-afcb-572c3e47407d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:12.012919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.012774 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg"] Nov 02 19:52:12.033339 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.031524 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-46f86d84\"" Nov 02 19:52:12.111151 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.111116 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfr9p\" (UniqueName: \"kubernetes.io/projected/60192f85-e356-4ca2-afcb-572c3e47407d-kube-api-access-rfr9p\") pod \"payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:12.111369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.111199 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60192f85-e356-4ca2-afcb-572c3e47407d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:12.111988 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.111967 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60192f85-e356-4ca2-afcb-572c3e47407d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:12.164379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.164343 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfr9p\" (UniqueName: \"kubernetes.io/projected/60192f85-e356-4ca2-afcb-572c3e47407d-kube-api-access-rfr9p\") pod \"payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:12.277758 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.277725 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:12.671603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:12.671215 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg"] Nov 02 19:52:13.327235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:13.327199 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" event={"ID":"60192f85-e356-4ca2-afcb-572c3e47407d","Type":"ContainerStarted","Data":"8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a"} Nov 02 19:52:13.327772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:13.327260 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" event={"ID":"60192f85-e356-4ca2-afcb-572c3e47407d","Type":"ContainerStarted","Data":"d6faa57814cf9b0dc2f2fd0f630db27c267722799a8a1e90583fba8b3318ff1e"} Nov 02 19:52:13.327772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:13.327336 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:13.443273 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:13.443212 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" podStartSLOduration=2.443197155 podStartE2EDuration="2.443197155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:13.438991839 +0000 UTC m=+1439.387647678" watchObservedRunningTime="2026-11-02 19:52:13.443197155 +0000 UTC m=+1439.391852993" Nov 02 19:52:14.776335 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:52:14.773373 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:52:14.911266 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.911224 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472"] Nov 02 19:52:14.917705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.917679 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:14.967120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.967054 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 19:52:14.967365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.967130 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 19:52:14.998574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.998465 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:14.998778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.998625 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:14.998778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.998710 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:14.998919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.998866 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vpw6\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-kube-api-access-2vpw6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:14.999044 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.999020 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea0192-0399-4102-b8de-da2709237b95-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:14.999160 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.999141 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:14.999242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:14.999223 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.026413 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.025524 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gd6zm\"" Nov 02 19:52:15.026413 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.025721 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 19:52:15.033464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.033435 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472"] Nov 02 19:52:15.100583 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.100544 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.100862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.100836 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.101005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.100988 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.101180 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.101161 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpw6\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-kube-api-access-2vpw6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.101361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.101342 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea0192-0399-4102-b8de-da2709237b95-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.101579 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.101559 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.101715 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.101697 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.107181 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.106650 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea0192-0399-4102-b8de-da2709237b95-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.109023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.108997 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.109672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.109652 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.109959 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.109938 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.111676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.111644 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.162525 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.161491 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpw6\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-kube-api-access-2vpw6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.169325 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.164061 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7q472\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.232082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.232052 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:15.716475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:15.716392 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472"] Nov 02 19:52:16.694468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:16.694411 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" event={"ID":"30ea0192-0399-4102-b8de-da2709237b95","Type":"ContainerStarted","Data":"8c0537539c4383d200431cee941854f66391ea000c1c26c1b44ecd4dc9383424"} Nov 02 19:52:21.332243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:21.332195 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:21.332631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:21.332266 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:21.769912 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:21.769860 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" event={"ID":"30ea0192-0399-4102-b8de-da2709237b95","Type":"ContainerStarted","Data":"5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89"} Nov 02 19:52:21.770121 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:21.769985 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:21.974979 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:21.974912 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" podStartSLOduration=2.403327485 podStartE2EDuration="7.974892353s" totalImagesPullingTime="5.571564868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:21.923710527 +0000 UTC m=+1447.872366367" watchObservedRunningTime="2026-11-02 19:52:21.974892353 +0000 UTC m=+1447.923548192" Nov 02 19:52:22.048423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:22.048344 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t"] Nov 02 19:52:22.054951 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:22.054915 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e12e5877-qzc5t"] Nov 02 19:52:22.187701 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:22.187667 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0e97b3-1653-4b94-a4cc-ea5e10e70668" path="/var/lib/kubelet/pods/8a0e97b3-1653-4b94-a4cc-ea5e10e70668/volumes" Nov 02 19:52:22.785601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:22.783418 4966 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7q472 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:52:22.785601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:22.783474 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" podUID="30ea0192-0399-4102-b8de-da2709237b95" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:52:24.469874 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:24.469841 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:52:25.783207 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:25.783167 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 19:52:44.753663 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:52:44.753618 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:52:47.049375 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:47.049337 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7"] Nov 02 19:52:47.094684 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:47.094649 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f1974b-ljvc7"] Nov 02 19:52:48.188248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:48.188211 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c625a86-1365-4c4b-8c72-17c4ce8d71af" path="/var/lib/kubelet/pods/4c625a86-1365-4c4b-8c72-17c4ce8d71af/volumes" Nov 02 19:52:50.069950 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.064763 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4"] Nov 02 19:52:50.080106 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.080078 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104238 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104483 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104353 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9z9\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-kube-api-access-cz9z9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104483 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104387 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104484 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104664 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104640 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104808 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104788 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104869 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104842 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb54a60-8aef-48f1-893c-daeca2a1f550-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.104927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104897 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.105040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.104932 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.167740 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.167702 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4"] Nov 02 19:52:50.207190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207150 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9z9\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-kube-api-access-cz9z9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207202 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207248 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207545 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207415 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207631 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207605 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207668 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207770 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207743 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb54a60-8aef-48f1-893c-daeca2a1f550-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207893 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207875 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.207963 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.207922 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.208055 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.208033 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.208505 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.208478 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.208897 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.208864 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb54a60-8aef-48f1-893c-daeca2a1f550-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.209235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.209209 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.209476 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.209451 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.210257 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.210232 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.211028 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.211008 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.260122 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.260096 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9z9\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-kube-api-access-cz9z9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.260284 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.260180 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f5bh4\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.403105 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.402967 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:50.845640 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.845597 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:50.845994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.845965 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:50.846340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.846315 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:50.847344 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:50.847275 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4"] Nov 02 19:52:51.086264 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:51.086157 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" event={"ID":"ecb54a60-8aef-48f1-893c-daeca2a1f550","Type":"ContainerStarted","Data":"7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae"} Nov 02 19:52:51.086264 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:51.086227 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" event={"ID":"ecb54a60-8aef-48f1-893c-daeca2a1f550","Type":"ContainerStarted","Data":"71530e69da5f22ae658f71382f0645298d500f95e65c0b35209213824ecfd8b2"} Nov 02 19:52:51.203784 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:51.203725 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podStartSLOduration=1.20371126 podStartE2EDuration="1.20371126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:51.197367886 +0000 UTC m=+1477.146023728" watchObservedRunningTime="2026-11-02 19:52:51.20371126 +0000 UTC m=+1477.152367099" Nov 02 19:52:51.403125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:51.403052 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:52:51.404632 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:51.404604 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:51.404772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:51.404656 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:52:52.403868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:52.403833 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:52.404416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:52.403893 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:52:53.403546 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:53.403506 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:53.403747 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:53.403565 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:52:54.403354 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:54.403283 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:54.403909 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:54.403373 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:52:55.065379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:55.065212 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn"] Nov 02 19:52:55.114056 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:55.114020 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f44951-gwfpn"] Nov 02 19:52:55.403515 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:55.403409 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:55.407908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:55.407862 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:52:56.223641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:56.223604 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565b1745-66c6-42ff-9252-afcca555a0eb" path="/var/lib/kubelet/pods/565b1745-66c6-42ff-9252-afcca555a0eb/volumes" Nov 02 19:52:56.405323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:56.404624 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:56.405323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:56.404687 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:52:57.066585 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:57.066547 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k"] Nov 02 19:52:57.092166 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:57.092130 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d15206-7vx9k"] Nov 02 19:52:57.408499 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:57.408416 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:57.408499 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:57.408481 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:52:58.188396 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:58.188359 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448a9154-e7f4-41fc-a523-ad07e46f4a75" path="/var/lib/kubelet/pods/448a9154-e7f4-41fc-a523-ad07e46f4a75/volumes" Nov 02 19:52:58.403901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:58.403859 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 19:52:58.404078 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:52:58.403933 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 19:53:00.403738 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:00.403690 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:53:00.404190 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:00.403761 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:53:01.444954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:01.444457 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:53:01.444954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:01.444544 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:53:02.045167 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:02.045129 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:53:02.045994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:02.045946 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:53:02.046895 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:02.046873 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 19:53:13.060643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:13.060590 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls"] Nov 02 19:53:13.124073 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:13.124033 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa2132e8-p8kls"] Nov 02 19:53:14.594316 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:14.594247 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f4fd51-e61d-445b-92ce-2304977e1892" path="/var/lib/kubelet/pods/37f4fd51-e61d-445b-92ce-2304977e1892/volumes" Nov 02 19:53:14.714687 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:14.714652 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:53:14.734685 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:14.734658 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:53:14.750331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:14.745928 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:53:14.785576 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:53:14.785536 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:53:14.788450 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:14.788425 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:53:16.747634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:16.747595 4966 scope.go:122] "RemoveContainer" containerID="c74f5485b5b6d2a9ad4eda8e37381c265a483b03ab63eff6acf8ad49511adf52" Nov 02 19:53:16.784511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:16.784485 4966 scope.go:122] "RemoveContainer" containerID="43ae5e6338cebd2d500754d2f359658cab663a4d31e772d9dd375e7b85c096b8" Nov 02 19:53:18.071317 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:18.070588 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64"] Nov 02 19:53:18.099047 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:18.099013 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44"] Nov 02 19:53:18.106992 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:18.106970 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1545a833-x9v64"] Nov 02 19:53:18.112125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:18.112100 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3694f937-tgr44"] Nov 02 19:53:18.185615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:18.185581 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676" path="/var/lib/kubelet/pods/1ea38265-8a8b-4ee0-b0eb-21ea6c6d8676/volumes" Nov 02 19:53:18.186124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:18.186109 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a418928-2d73-43ca-a1cd-1eed624b7391" path="/var/lib/kubelet/pods/6a418928-2d73-43ca-a1cd-1eed624b7391/volumes" Nov 02 19:53:20.434646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:20.434604 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7"] Nov 02 19:53:20.435502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:20.435464 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" containerID="cri-o://89b1c6b01f6f80d33744534f8489f35d23658d48a24dfe66b087080c34c6dc4f" gracePeriod=30 Nov 02 19:53:25.705054 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.703846 4966 generic.go:356] "Generic (PLEG): container finished" podID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerID="89b1c6b01f6f80d33744534f8489f35d23658d48a24dfe66b087080c34c6dc4f" exitCode=0 Nov 02 19:53:25.709543 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.709439 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" event={"ID":"a253b41d-9ef0-4447-a246-fd2579ef1d5f","Type":"ContainerDied","Data":"89b1c6b01f6f80d33744534f8489f35d23658d48a24dfe66b087080c34c6dc4f"} Nov 02 19:53:25.714478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.713274 4966 generic.go:356] "Generic (PLEG): container finished" podID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerID="c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb" exitCode=0 Nov 02 19:53:25.714478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.713421 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" event={"ID":"5eb3fa1b-d973-4e78-8340-a204313c171f","Type":"ContainerDied","Data":"c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb"} Nov 02 19:53:25.714478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.714048 4966 scope.go:122] "RemoveContainer" containerID="c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb" Nov 02 19:53:25.719755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.718790 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:53:25.719755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.718884 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:53:25.719755 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.718926 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:53:25.862875 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:25.862787 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:53:26.131423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.128012 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb"] Nov 02 19:53:26.151783 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.151747 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5c625c1-gp9hb"] Nov 02 19:53:26.224149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.224104 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e4ff5c-2443-45aa-afa4-83300ac39fc9" path="/var/lib/kubelet/pods/e5e4ff5c-2443-45aa-afa4-83300ac39fc9/volumes" Nov 02 19:53:26.504883 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.504850 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:53:26.531895 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.531862 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-certs\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532110 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532080 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istiod-ca-cert\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532119 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-podinfo\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532127 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-certs" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.532214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532153 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-kube-api-access-rsfb6\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-kube-api-access-rsfb6\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532189 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-socket\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532261 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-token\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-token\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532310 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-envoy\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532374 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-credential-socket\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532407 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-data\") pod \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\" (UID: \"a253b41d-9ef0-4447-a246-fd2579ef1d5f\") " Nov 02 19:53:26.532720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532537 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istiod-ca-cert" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:26.532810 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532783 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-credential-socket" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.532884 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532806 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-data" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.532884 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.532861 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-socket" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.533199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.533177 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.533331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.533203 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.533331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.533220 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.533331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.533236 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.533331 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.533250 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.534605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.534574 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-podinfo" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:53:26.534899 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.534875 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-token" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:26.534996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.534917 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-kube-api-access-rsfb6" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "kube-api-access-rsfb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:26.535194 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.535173 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-envoy" pod "a253b41d-9ef0-4447-a246-fd2579ef1d5f" (UID: "a253b41d-9ef0-4447-a246-fd2579ef1d5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:26.635228 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.635142 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.635228 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.635176 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsfb6\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-kube-api-access-rsfb6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.635228 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.635188 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.635228 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.635197 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a253b41d-9ef0-4447-a246-fd2579ef1d5f-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:26.732807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.732165 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" Nov 02 19:53:26.732807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.732179 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7" event={"ID":"a253b41d-9ef0-4447-a246-fd2579ef1d5f","Type":"ContainerDied","Data":"9045a144c9f14ef74c598117cb68bb35f16c092bda5a45c8671e0d15f0921f4d"} Nov 02 19:53:26.732807 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.732246 4966 scope.go:122] "RemoveContainer" containerID="89b1c6b01f6f80d33744534f8489f35d23658d48a24dfe66b087080c34c6dc4f" Nov 02 19:53:26.740781 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:26.739855 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" event={"ID":"5eb3fa1b-d973-4e78-8340-a204313c171f","Type":"ContainerStarted","Data":"4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170"} Nov 02 19:53:27.030859 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:27.030823 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7"] Nov 02 19:53:27.069318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:27.068421 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-fmlj7"] Nov 02 19:53:27.716006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:27.715962 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:53:27.716890 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:27.716861 4966 patch_prober.go:29] interesting pod/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 19:53:27.717013 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:27.716931 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 19:53:28.068246 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:28.068204 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5"] Nov 02 19:53:28.075368 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:28.075338 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ede02fc0-cjtg5"] Nov 02 19:53:28.188006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:28.187971 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947f465c-d8b4-4b83-af9f-28190d4a3888" path="/var/lib/kubelet/pods/947f465c-d8b4-4b83-af9f-28190d4a3888/volumes" Nov 02 19:53:28.188601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:28.188584 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" path="/var/lib/kubelet/pods/a253b41d-9ef0-4447-a246-fd2579ef1d5f/volumes" Nov 02 19:53:28.716419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:28.716291 4966 patch_prober.go:29] interesting pod/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 19:53:28.716643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:28.716509 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 19:53:29.068390 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:29.068346 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk"] Nov 02 19:53:29.110363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:29.110327 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0d71fd-98gfk"] Nov 02 19:53:29.716087 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:29.716047 4966 patch_prober.go:29] interesting pod/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 19:53:29.716280 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:29.716108 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 19:53:30.190492 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:30.190453 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b46edcb-15a4-4da0-b24b-6c5d3309194e" path="/var/lib/kubelet/pods/1b46edcb-15a4-4da0-b24b-6c5d3309194e/volumes" Nov 02 19:53:30.716210 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:30.716173 4966 patch_prober.go:29] interesting pod/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 19:53:30.716413 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:30.716238 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 19:53:31.067914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:31.067870 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx"] Nov 02 19:53:31.096939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:31.096903 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c948dd00-ww9xx"] Nov 02 19:53:31.716734 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:31.716697 4966 patch_prober.go:29] interesting pod/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 19:53:31.717130 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:31.716755 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 19:53:32.185646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:32.185614 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26757aed-c515-4053-91ce-bfc01caba2c1" path="/var/lib/kubelet/pods/26757aed-c515-4053-91ce-bfc01caba2c1/volumes" Nov 02 19:53:32.724824 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:32.724780 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:53:32.727260 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:32.726608 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:53:32.727491 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:32.727469 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:53:40.050098 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.050050 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b"] Nov 02 19:53:40.093227 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.093194 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eaef11f-7sq6b"] Nov 02 19:53:40.187722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.187691 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a4bf4e-872e-4e49-8d7c-a1f418210eef" path="/var/lib/kubelet/pods/78a4bf4e-872e-4e49-8d7c-a1f418210eef/volumes" Nov 02 19:53:40.552319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.536226 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n"] Nov 02 19:53:40.552319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.537175 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" Nov 02 19:53:40.552319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.537194 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" Nov 02 19:53:40.558625 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.558588 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a253b41d-9ef0-4447-a246-fd2579ef1d5f" containerName="istio-proxy" Nov 02 19:53:40.563460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.563437 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.593345 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.593313 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n"] Nov 02 19:53:40.629473 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.629425 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjwqc\" (UniqueName: \"kubernetes.io/projected/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-kube-api-access-kjwqc\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.629663 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.629588 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.629724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.629704 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.632571 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.632541 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:53:40.632700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.632599 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:53:40.632700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.632599 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:53:40.632825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.632756 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:53:40.730641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.730601 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.730820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.730665 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.730820 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.730815 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwqc\" (UniqueName: \"kubernetes.io/projected/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-kube-api-access-kjwqc\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.731477 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.731456 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.733006 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.732985 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.784035 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.783997 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwqc\" (UniqueName: \"kubernetes.io/projected/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-kube-api-access-kjwqc\") pod \"maas-api-revoke-keys-e2e-ait-d368a053-p548n\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:40.878717 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:40.878620 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:41.264603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:41.264576 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n"] Nov 02 19:53:41.925216 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:41.925112 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" event={"ID":"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e","Type":"ContainerStarted","Data":"0e7dfa90c5492f05d8073908df06245a64b55f9d1897863470f4a711d1ae3116"} Nov 02 19:53:41.925216 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:41.925178 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" event={"ID":"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e","Type":"ContainerStarted","Data":"6a776643fcc4247ea6c769808e3c76bf23557f43ed982370ee1757d7fb3c4f7b"} Nov 02 19:53:42.033087 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:42.033020 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" podStartSLOduration=2.033002179 podStartE2EDuration="2.033002179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:42.030027286 +0000 UTC m=+1527.978683117" watchObservedRunningTime="2026-11-02 19:53:42.033002179 +0000 UTC m=+1527.981658019" Nov 02 19:53:42.082710 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:42.082671 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb"] Nov 02 19:53:42.087587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:42.087559 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-108cbcef-6pscb"] Nov 02 19:53:42.189521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:42.189450 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e92616f-1c77-4ad5-9f8d-12f61183fa1c" path="/var/lib/kubelet/pods/7e92616f-1c77-4ad5-9f8d-12f61183fa1c/volumes" Nov 02 19:53:42.935520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:42.935483 4966 generic.go:356] "Generic (PLEG): container finished" podID="efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" containerID="0e7dfa90c5492f05d8073908df06245a64b55f9d1897863470f4a711d1ae3116" exitCode=0 Nov 02 19:53:42.935966 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:42.935567 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" event={"ID":"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e","Type":"ContainerDied","Data":"0e7dfa90c5492f05d8073908df06245a64b55f9d1897863470f4a711d1ae3116"} Nov 02 19:53:44.147341 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.147275 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:44.195257 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.195179 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-xks-ca\") pod \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " Nov 02 19:53:44.195438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.195359 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-kube-api-access-kjwqc\" (UniqueName: \"kubernetes.io/projected/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-kube-api-access-kjwqc\") pod \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " Nov 02 19:53:44.195662 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.195638 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-openshift-ca\") pod \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\" (UID: \"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e\") " Nov 02 19:53:44.195996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.195973 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-openshift-ca" pod "efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" (UID: "efc8dcf3-579b-4edf-9701-4a4b7b6ab93e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:44.196376 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.196357 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:44.197468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.197439 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-xks-ca" pod "efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" (UID: "efc8dcf3-579b-4edf-9701-4a4b7b6ab93e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:44.197678 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.197656 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-kube-api-access-kjwqc" pod "efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" (UID: "efc8dcf3-579b-4edf-9701-4a4b7b6ab93e"). InnerVolumeSpecName "kube-api-access-kjwqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:44.298575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.298528 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:44.298575 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.298574 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjwqc\" (UniqueName: \"kubernetes.io/projected/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e-kube-api-access-kjwqc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:44.755424 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:53:44.755379 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:53:44.956725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.956691 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" Nov 02 19:53:44.956888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.956724 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n" event={"ID":"efc8dcf3-579b-4edf-9701-4a4b7b6ab93e","Type":"ContainerDied","Data":"6a776643fcc4247ea6c769808e3c76bf23557f43ed982370ee1757d7fb3c4f7b"} Nov 02 19:53:44.956888 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:44.956814 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a776643fcc4247ea6c769808e3c76bf23557f43ed982370ee1757d7fb3c4f7b" Nov 02 19:53:50.170516 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.170478 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c"] Nov 02 19:53:50.171717 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.171694 4966 cpu_manager.go:414] "containerMap: removing container" podUID="efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" containerName="revoke-keys" Nov 02 19:53:50.171787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.171720 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" containerName="revoke-keys" Nov 02 19:53:50.171936 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.171924 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" containerName="revoke-keys" Nov 02 19:53:50.176736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.176703 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.226529 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.226484 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c"] Nov 02 19:53:50.227338 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.227275 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:53:50.266125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.266091 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:53:50.273436 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.273407 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:53:50.273609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.273465 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:53:50.311613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.311574 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.311876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.311851 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.312094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.312069 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp99t\" (UniqueName: \"kubernetes.io/projected/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-kube-api-access-dp99t\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.413796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.413751 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.414003 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.413925 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.414056 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.414023 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp99t\" (UniqueName: \"kubernetes.io/projected/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-kube-api-access-dp99t\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.414628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.414602 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.416636 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.416605 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.466930 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.466863 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp99t\" (UniqueName: \"kubernetes.io/projected/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-kube-api-access-dp99t\") pod \"maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.493252 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.493223 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:50.873882 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:50.873850 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c"] Nov 02 19:53:50.874556 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:53:50.874519 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe26d6a7_dceb_47a2_b4e6_72833c9c646c.slice/crio-794201ed880a5c57cf97bae2a328decb9f5f7556752e29583f25efd0a3f4e567 WatchSource:0}: Error finding container 794201ed880a5c57cf97bae2a328decb9f5f7556752e29583f25efd0a3f4e567: Status 404 returned error can't find the container with id 794201ed880a5c57cf97bae2a328decb9f5f7556752e29583f25efd0a3f4e567 Nov 02 19:53:51.033318 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:51.033270 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" event={"ID":"fe26d6a7-dceb-47a2-b4e6-72833c9c646c","Type":"ContainerStarted","Data":"794201ed880a5c57cf97bae2a328decb9f5f7556752e29583f25efd0a3f4e567"} Nov 02 19:53:52.049218 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:52.049173 4966 generic.go:356] "Generic (PLEG): container finished" podID="fe26d6a7-dceb-47a2-b4e6-72833c9c646c" containerID="671bf852f9daf7be7f5dd3a039ccb9dedbde4f735a2d58068e7e27c372735035" exitCode=0 Nov 02 19:53:52.049720 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:52.049252 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" event={"ID":"fe26d6a7-dceb-47a2-b4e6-72833c9c646c","Type":"ContainerDied","Data":"671bf852f9daf7be7f5dd3a039ccb9dedbde4f735a2d58068e7e27c372735035"} Nov 02 19:53:53.510320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.509606 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:53.601321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.600981 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-xks-ca\") pod \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " Nov 02 19:53:53.601321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.601062 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-openshift-ca\") pod \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " Nov 02 19:53:53.601321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.601283 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-kube-api-access-dp99t\" (UniqueName: \"kubernetes.io/projected/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-kube-api-access-dp99t\") pod \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\" (UID: \"fe26d6a7-dceb-47a2-b4e6-72833c9c646c\") " Nov 02 19:53:53.617324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.603126 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-openshift-ca" pod "fe26d6a7-dceb-47a2-b4e6-72833c9c646c" (UID: "fe26d6a7-dceb-47a2-b4e6-72833c9c646c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:53.663506 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.652737 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-xks-ca" pod "fe26d6a7-dceb-47a2-b4e6-72833c9c646c" (UID: "fe26d6a7-dceb-47a2-b4e6-72833c9c646c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:53.672370 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.669486 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-kube-api-access-dp99t" pod "fe26d6a7-dceb-47a2-b4e6-72833c9c646c" (UID: "fe26d6a7-dceb-47a2-b4e6-72833c9c646c"). InnerVolumeSpecName "kube-api-access-dp99t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:53.797372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.793638 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:53.797372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.793675 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:53.797372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:53.793691 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp99t\" (UniqueName: \"kubernetes.io/projected/fe26d6a7-dceb-47a2-b4e6-72833c9c646c-kube-api-access-dp99t\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:54.126321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:54.125999 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" event={"ID":"fe26d6a7-dceb-47a2-b4e6-72833c9c646c","Type":"ContainerDied","Data":"794201ed880a5c57cf97bae2a328decb9f5f7556752e29583f25efd0a3f4e567"} Nov 02 19:53:54.126321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:54.126079 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794201ed880a5c57cf97bae2a328decb9f5f7556752e29583f25efd0a3f4e567" Nov 02 19:53:54.126321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:54.126173 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c" Nov 02 19:53:59.047193 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:59.047156 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m"] Nov 02 19:53:59.092714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:53:59.092682 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341acaec-nqp6m"] Nov 02 19:54:00.046672 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:00.046626 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8"] Nov 02 19:54:00.092255 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:00.092219 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d5981b-dh9f8"] Nov 02 19:54:00.187287 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:00.187248 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8536d875-ad0d-4a5c-bdb0-55720314e10e" path="/var/lib/kubelet/pods/8536d875-ad0d-4a5c-bdb0-55720314e10e/volumes" Nov 02 19:54:00.187829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:00.187805 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6ef62a-982c-47d9-85b0-c1f4873013a8" path="/var/lib/kubelet/pods/8c6ef62a-982c-47d9-85b0-c1f4873013a8/volumes" Nov 02 19:54:01.045471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:01.045432 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq"] Nov 02 19:54:01.091328 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:01.091282 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe3ac97c-5mrmq"] Nov 02 19:54:01.738603 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:01.738402 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg"] Nov 02 19:54:02.190112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:02.190032 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e548f9-d865-4521-827a-e2168ed336c7" path="/var/lib/kubelet/pods/60e548f9-d865-4521-827a-e2168ed336c7/volumes" Nov 02 19:54:06.077484 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:06.077444 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks"] Nov 02 19:54:06.112970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:06.112933 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25fdc4fd-64cks"] Nov 02 19:54:06.187890 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:06.187854 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67118bc7-f99c-47f8-894d-8d82423e1435" path="/var/lib/kubelet/pods/67118bc7-f99c-47f8-894d-8d82423e1435/volumes" Nov 02 19:54:06.740005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:06.739951 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" podUID="60192f85-e356-4ca2-afcb-572c3e47407d" containerName="payload-pre-processing" containerID="cri-o://8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a" gracePeriod=25 Nov 02 19:54:07.056658 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.056629 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:54:07.176397 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.176362 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60192f85-e356-4ca2-afcb-572c3e47407d-kube-api-access-rfr9p\" (UniqueName: \"kubernetes.io/projected/60192f85-e356-4ca2-afcb-572c3e47407d-kube-api-access-rfr9p\") pod \"60192f85-e356-4ca2-afcb-572c3e47407d\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " Nov 02 19:54:07.176858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.176533 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60192f85-e356-4ca2-afcb-572c3e47407d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60192f85-e356-4ca2-afcb-572c3e47407d-plugins-config-volume\") pod \"60192f85-e356-4ca2-afcb-572c3e47407d\" (UID: \"60192f85-e356-4ca2-afcb-572c3e47407d\") " Nov 02 19:54:07.176934 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.176891 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60192f85-e356-4ca2-afcb-572c3e47407d-plugins-config-volume" pod "60192f85-e356-4ca2-afcb-572c3e47407d" (UID: "60192f85-e356-4ca2-afcb-572c3e47407d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:07.177256 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.177224 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60192f85-e356-4ca2-afcb-572c3e47407d-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:07.178900 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.178858 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60192f85-e356-4ca2-afcb-572c3e47407d-kube-api-access-rfr9p" pod "60192f85-e356-4ca2-afcb-572c3e47407d" (UID: "60192f85-e356-4ca2-afcb-572c3e47407d"). InnerVolumeSpecName "kube-api-access-rfr9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:07.279583 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.279478 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfr9p\" (UniqueName: \"kubernetes.io/projected/60192f85-e356-4ca2-afcb-572c3e47407d-kube-api-access-rfr9p\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:07.512868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.512836 4966 generic.go:356] "Generic (PLEG): container finished" podID="60192f85-e356-4ca2-afcb-572c3e47407d" containerID="8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a" exitCode=0 Nov 02 19:54:07.513037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.512893 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" Nov 02 19:54:07.513037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.512893 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" event={"ID":"60192f85-e356-4ca2-afcb-572c3e47407d","Type":"ContainerDied","Data":"8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a"} Nov 02 19:54:07.513037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.512959 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg" event={"ID":"60192f85-e356-4ca2-afcb-572c3e47407d","Type":"ContainerDied","Data":"d6faa57814cf9b0dc2f2fd0f630db27c267722799a8a1e90583fba8b3318ff1e"} Nov 02 19:54:07.513037 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.512990 4966 scope.go:122] "RemoveContainer" containerID="8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a" Nov 02 19:54:07.525425 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.525405 4966 scope.go:122] "RemoveContainer" containerID="8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a" Nov 02 19:54:07.525697 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:54:07.525677 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a\": container with ID starting with 8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a not found: ID does not exist" containerID="8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a" Nov 02 19:54:07.525772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.525704 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a"} err="failed to get container status \"8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a\": rpc error: code = NotFound desc = could not find container \"8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a\": container with ID starting with 8839cc3213af4658c8dbedf9444f7ba72b8c056d3568503444e821a202f0678a not found: ID does not exist" Nov 02 19:54:07.668454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.668361 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg"] Nov 02 19:54:07.714676 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:07.714638 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46f86d84-5df57c4ddc-wztpg"] Nov 02 19:54:08.186214 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:08.186184 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60192f85-e356-4ca2-afcb-572c3e47407d" path="/var/lib/kubelet/pods/60192f85-e356-4ca2-afcb-572c3e47407d/volumes" Nov 02 19:54:09.049714 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:09.049663 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn"] Nov 02 19:54:09.091847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:09.091811 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b42fa7-zw8zn"] Nov 02 19:54:10.187884 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:10.187847 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fd34c6-64ed-4083-a4ac-f8dfa279fbe1" path="/var/lib/kubelet/pods/20fd34c6-64ed-4083-a4ac-f8dfa279fbe1/volumes" Nov 02 19:54:12.047680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:12.047643 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj"] Nov 02 19:54:12.091146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:12.091113 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dbe1dd6-vwzxj"] Nov 02 19:54:12.186467 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:12.186424 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefd1c16-7116-4f45-af0d-49b44fdba4f8" path="/var/lib/kubelet/pods/eefd1c16-7116-4f45-af0d-49b44fdba4f8/volumes" Nov 02 19:54:13.046747 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:13.046698 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g"] Nov 02 19:54:13.090049 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:13.090011 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7e4a37-zwc9g"] Nov 02 19:54:14.048503 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:14.048463 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x"] Nov 02 19:54:14.092833 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:14.092795 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2748ff-qlw5x"] Nov 02 19:54:14.186625 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:14.186594 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54557b95-d415-4f48-a5e1-63cc56569d31" path="/var/lib/kubelet/pods/54557b95-d415-4f48-a5e1-63cc56569d31/volumes" Nov 02 19:54:14.187124 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:14.187110 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae" path="/var/lib/kubelet/pods/aa08fb5d-e3bd-45c4-af7d-a54fbd1499ae/volumes" Nov 02 19:54:14.759863 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:54:14.759813 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:54:15.045526 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:15.045441 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr"] Nov 02 19:54:15.091440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:15.091406 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6742d7-4jjzr"] Nov 02 19:54:16.185850 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.185817 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7697a9-283a-44f0-9f4a-7e456dcea93e" path="/var/lib/kubelet/pods/ce7697a9-283a-44f0-9f4a-7e456dcea93e/volumes" Nov 02 19:54:16.878972 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.878935 4966 scope.go:122] "RemoveContainer" containerID="53fc8dfde2e5e95dc43ac6c7b60090304854e6580527160808bc0a6cb466f225" Nov 02 19:54:16.890924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.890902 4966 scope.go:122] "RemoveContainer" containerID="d4b64393ba0cdefa97fc729b4f8c65982ea0b875bfb7a21d8a85763c8421e938" Nov 02 19:54:16.901856 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.901835 4966 scope.go:122] "RemoveContainer" containerID="9038d58cbae70d40e08d70797f7c1b6b9f39e0ff69c25bd44fff625f62fca272" Nov 02 19:54:16.913459 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.913428 4966 scope.go:122] "RemoveContainer" containerID="753703e9fee0f505ad5d757cad10103f5a2ca698cf53e63edeaafd32047a4c5c" Nov 02 19:54:16.923995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.923974 4966 scope.go:122] "RemoveContainer" containerID="a0ee2ae818fbb3e115db770f8d1c36e75b5ca41507e6a5c6b8edd8a49f16df4b" Nov 02 19:54:16.934379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.934362 4966 scope.go:122] "RemoveContainer" containerID="e10f31932b251766c8f1115ae9155f8b5816ef5dfef77ee4c8bf3b75be207874" Nov 02 19:54:16.944417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:16.944399 4966 scope.go:122] "RemoveContainer" containerID="2baa328b432229497eda8f81629c51ff9ba8550b4ef5b6c787895ff163ca64ce" Nov 02 19:54:17.830199 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.830164 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t"] Nov 02 19:54:17.830921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.830901 4966 cpu_manager.go:414] "containerMap: removing container" podUID="fe26d6a7-dceb-47a2-b4e6-72833c9c646c" containerName="revoke-keys" Nov 02 19:54:17.830974 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.830922 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe26d6a7-dceb-47a2-b4e6-72833c9c646c" containerName="revoke-keys" Nov 02 19:54:17.830974 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.830947 4966 cpu_manager.go:414] "containerMap: removing container" podUID="60192f85-e356-4ca2-afcb-572c3e47407d" containerName="payload-pre-processing" Nov 02 19:54:17.830974 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.830952 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60192f85-e356-4ca2-afcb-572c3e47407d" containerName="payload-pre-processing" Nov 02 19:54:17.831080 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.831071 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe26d6a7-dceb-47a2-b4e6-72833c9c646c" containerName="revoke-keys" Nov 02 19:54:17.831115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.831085 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="60192f85-e356-4ca2-afcb-572c3e47407d" containerName="payload-pre-processing" Nov 02 19:54:17.836854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.836833 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:17.887927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.887892 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t"] Nov 02 19:54:17.928061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.928024 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:54:17.928061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.928030 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:54:17.928061 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.928039 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:54:17.930821 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.930800 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:54:17.948012 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.947970 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:17.948419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.948141 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:17.948419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:17.948286 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smwf\" (UniqueName: \"kubernetes.io/projected/96100327-a10a-4fcf-a9e0-e7457db3a986-kube-api-access-6smwf\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.049507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.049460 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.049731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.049551 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.049731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.049623 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smwf\" (UniqueName: \"kubernetes.io/projected/96100327-a10a-4fcf-a9e0-e7457db3a986-kube-api-access-6smwf\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.050377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.050347 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.052256 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.052228 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.102217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.102145 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smwf\" (UniqueName: \"kubernetes.io/projected/96100327-a10a-4fcf-a9e0-e7457db3a986-kube-api-access-6smwf\") pod \"maas-api-revoke-sub-774023d1-gq84t\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.151156 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.151122 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:18.525368 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:54:18.525332 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96100327_a10a_4fcf_a9e0_e7457db3a986.slice/crio-ae16d8488d563ee5adf5d83810b3f93985e193cc2464fa4e7d4dc717f55052b1 WatchSource:0}: Error finding container ae16d8488d563ee5adf5d83810b3f93985e193cc2464fa4e7d4dc717f55052b1: Status 404 returned error can't find the container with id ae16d8488d563ee5adf5d83810b3f93985e193cc2464fa4e7d4dc717f55052b1 Nov 02 19:54:18.526220 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.526196 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t"] Nov 02 19:54:18.629512 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:18.629463 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" event={"ID":"96100327-a10a-4fcf-a9e0-e7457db3a986","Type":"ContainerStarted","Data":"ae16d8488d563ee5adf5d83810b3f93985e193cc2464fa4e7d4dc717f55052b1"} Nov 02 19:54:19.640660 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:19.640625 4966 generic.go:356] "Generic (PLEG): container finished" podID="96100327-a10a-4fcf-a9e0-e7457db3a986" containerID="b3c5b6d64727eb8d339a8918aa4154aff97210d94610cf388ef92ee5b96da09d" exitCode=0 Nov 02 19:54:19.641033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:19.640708 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" event={"ID":"96100327-a10a-4fcf-a9e0-e7457db3a986","Type":"ContainerDied","Data":"b3c5b6d64727eb8d339a8918aa4154aff97210d94610cf388ef92ee5b96da09d"} Nov 02 19:54:20.837934 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.837909 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:20.901405 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.901323 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96100327-a10a-4fcf-a9e0-e7457db3a986-kube-api-access-6smwf\" (UniqueName: \"kubernetes.io/projected/96100327-a10a-4fcf-a9e0-e7457db3a986-kube-api-access-6smwf\") pod \"96100327-a10a-4fcf-a9e0-e7457db3a986\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " Nov 02 19:54:20.901563 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.901533 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-openshift-ca\") pod \"96100327-a10a-4fcf-a9e0-e7457db3a986\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " Nov 02 19:54:20.901658 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.901643 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-xks-ca\") pod \"96100327-a10a-4fcf-a9e0-e7457db3a986\" (UID: \"96100327-a10a-4fcf-a9e0-e7457db3a986\") " Nov 02 19:54:20.901905 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.901883 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-openshift-ca" pod "96100327-a10a-4fcf-a9e0-e7457db3a986" (UID: "96100327-a10a-4fcf-a9e0-e7457db3a986"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:20.902346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.902328 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:20.903582 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.903563 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96100327-a10a-4fcf-a9e0-e7457db3a986-kube-api-access-6smwf" pod "96100327-a10a-4fcf-a9e0-e7457db3a986" (UID: "96100327-a10a-4fcf-a9e0-e7457db3a986"). InnerVolumeSpecName "kube-api-access-6smwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:20.903659 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:20.903625 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-xks-ca" pod "96100327-a10a-4fcf-a9e0-e7457db3a986" (UID: "96100327-a10a-4fcf-a9e0-e7457db3a986"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:54:21.004195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:21.004157 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96100327-a10a-4fcf-a9e0-e7457db3a986-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:21.004195 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:21.004193 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smwf\" (UniqueName: \"kubernetes.io/projected/96100327-a10a-4fcf-a9e0-e7457db3a986-kube-api-access-6smwf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:21.660149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:21.660121 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" Nov 02 19:54:21.660342 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:21.660157 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t" event={"ID":"96100327-a10a-4fcf-a9e0-e7457db3a986","Type":"ContainerDied","Data":"ae16d8488d563ee5adf5d83810b3f93985e193cc2464fa4e7d4dc717f55052b1"} Nov 02 19:54:21.660342 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:21.660207 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae16d8488d563ee5adf5d83810b3f93985e193cc2464fa4e7d4dc717f55052b1" Nov 02 19:54:23.047162 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:23.047122 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x"] Nov 02 19:54:23.093688 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:23.093650 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cd1fcc-btr9x"] Nov 02 19:54:23.098309 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:23.098266 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2"] Nov 02 19:54:23.102775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:23.102750 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434d6db5-94gm2"] Nov 02 19:54:24.185555 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:24.185523 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f20784f-9892-49f3-b784-c2837fa2afdf" path="/var/lib/kubelet/pods/8f20784f-9892-49f3-b784-c2837fa2afdf/volumes" Nov 02 19:54:24.186133 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:24.186116 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c970045e-af95-4c6b-be41-065f4837caaa" path="/var/lib/kubelet/pods/c970045e-af95-4c6b-be41-065f4837caaa/volumes" Nov 02 19:54:25.041890 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:25.041809 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm"] Nov 02 19:54:25.087724 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:25.087692 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8437d0d-jtrtm"] Nov 02 19:54:26.187731 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:26.187699 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bb2374-6d61-4104-8112-2196ccb80fe7" path="/var/lib/kubelet/pods/37bb2374-6d61-4104-8112-2196ccb80fe7/volumes" Nov 02 19:54:31.047763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:31.047726 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj"] Nov 02 19:54:31.091204 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:31.091168 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c7a7495-qk6rj"] Nov 02 19:54:31.892225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:31.892187 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n"] Nov 02 19:54:31.937679 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:31.937641 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d368a053-p548n"] Nov 02 19:54:32.188906 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:32.188813 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb668698-f97a-4c99-84ed-11c256d34fdf" path="/var/lib/kubelet/pods/cb668698-f97a-4c99-84ed-11c256d34fdf/volumes" Nov 02 19:54:32.189504 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:32.189485 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc8dcf3-579b-4edf-9701-4a4b7b6ab93e" path="/var/lib/kubelet/pods/efc8dcf3-579b-4edf-9701-4a4b7b6ab93e/volumes" Nov 02 19:54:33.404140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:33.404101 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7"] Nov 02 19:54:33.404651 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:33.404373 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" containerID="cri-o://4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170" gracePeriod=30 Nov 02 19:54:38.020094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.020060 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c"] Nov 02 19:54:38.065491 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.065454 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46f86d84-wsf2c"] Nov 02 19:54:38.231643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.231598 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe26d6a7-dceb-47a2-b4e6-72833c9c646c" path="/var/lib/kubelet/pods/fe26d6a7-dceb-47a2-b4e6-72833c9c646c/volumes" Nov 02 19:54:38.715018 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.714988 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:54:38.821740 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.821701 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-certs\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.821966 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.821792 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-kube-api-access-s5gv4\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-kube-api-access-s5gv4\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.821966 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.821860 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-data\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.821966 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.821894 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-token\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.822423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.822219 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-data" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:38.822423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.821921 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-envoy\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.822423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.822319 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-certs" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:38.822622 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.822584 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-socket\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.822712 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.822695 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-podinfo\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.822773 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.822738 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb3fa1b-d973-4e78-8340-a204313c171f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3fa1b-d973-4e78-8340-a204313c171f-istiod-ca-cert\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.822815 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.822769 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-credential-socket\") pod \"5eb3fa1b-d973-4e78-8340-a204313c171f\" (UID: \"5eb3fa1b-d973-4e78-8340-a204313c171f\") " Nov 02 19:54:38.822968 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.822942 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-socket" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:38.823263 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.823209 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-credential-socket" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:38.823417 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.823391 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb3fa1b-d973-4e78-8340-a204313c171f-istiod-ca-cert" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:38.823568 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.823544 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.823568 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.823570 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb3fa1b-d973-4e78-8340-a204313c171f-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.823722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.823584 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.823722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.823597 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.823722 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.823608 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.824457 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.824439 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-token" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:38.824705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.824682 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-envoy" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:38.824792 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.824744 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-kube-api-access-s5gv4" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "kube-api-access-s5gv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:38.825026 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.825011 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-podinfo" pod "5eb3fa1b-d973-4e78-8340-a204313c171f" (UID: "5eb3fa1b-d973-4e78-8340-a204313c171f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:54:38.925245 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.925209 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5gv4\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-kube-api-access-s5gv4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.925245 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.925242 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.925245 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.925252 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.925520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.925262 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb3fa1b-d973-4e78-8340-a204313c171f-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:38.935501 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.935426 4966 generic.go:356] "Generic (PLEG): container finished" podID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerID="4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170" exitCode=0 Nov 02 19:54:38.935501 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.935478 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" Nov 02 19:54:38.935694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.935499 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" event={"ID":"5eb3fa1b-d973-4e78-8340-a204313c171f","Type":"ContainerDied","Data":"4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170"} Nov 02 19:54:38.935694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.935551 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7" event={"ID":"5eb3fa1b-d973-4e78-8340-a204313c171f","Type":"ContainerDied","Data":"f23872dc62b8efc224e14b781a13cad3e2293ffea2c01189d6d974dc40a430c6"} Nov 02 19:54:38.935694 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.935573 4966 scope.go:122] "RemoveContainer" containerID="4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170" Nov 02 19:54:38.948538 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.948516 4966 scope.go:122] "RemoveContainer" containerID="c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb" Nov 02 19:54:38.962493 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.962467 4966 scope.go:122] "RemoveContainer" containerID="4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170" Nov 02 19:54:38.962830 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:54:38.962808 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170\": container with ID starting with 4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170 not found: ID does not exist" containerID="4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170" Nov 02 19:54:38.962956 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.962843 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170"} err="failed to get container status \"4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170\": rpc error: code = NotFound desc = could not find container \"4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170\": container with ID starting with 4c12a5aae7de093900100e8c41c8e4bbb16cb452f2115c7f321ffb6e061be170 not found: ID does not exist" Nov 02 19:54:38.962956 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.962873 4966 scope.go:122] "RemoveContainer" containerID="c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb" Nov 02 19:54:38.963379 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:54:38.963353 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb\": container with ID starting with c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb not found: ID does not exist" containerID="c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb" Nov 02 19:54:38.963475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:38.963389 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb"} err="failed to get container status \"c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb\": rpc error: code = NotFound desc = could not find container \"c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb\": container with ID starting with c93d5a549f0c2967234af91307b60e5209f067d8207bde658a21e1b13a8badcb not found: ID does not exist" Nov 02 19:54:39.092246 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:39.092206 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7"] Nov 02 19:54:39.100398 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:39.098922 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d368a053-openshift-default-85bcfb7c57-hqqj7"] Nov 02 19:54:40.190048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:40.190010 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" path="/var/lib/kubelet/pods/5eb3fa1b-d973-4e78-8340-a204313c171f/volumes" Nov 02 19:54:43.050023 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:43.049988 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl"] Nov 02 19:54:43.093229 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:43.093180 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1863c658-ngrgl"] Nov 02 19:54:44.193050 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:44.192644 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3343608e-2dbd-4473-a794-0e96a7169300" path="/var/lib/kubelet/pods/3343608e-2dbd-4473-a794-0e96a7169300/volumes" Nov 02 19:54:44.755971 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:54:44.755925 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:54:51.044732 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:51.044688 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7"] Nov 02 19:54:51.090768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:51.090732 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4df6b7-t8zh7"] Nov 02 19:54:52.201069 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:52.201038 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e01470-20f7-4ded-ad9b-63ec1848cf5d" path="/var/lib/kubelet/pods/92e01470-20f7-4ded-ad9b-63ec1848cf5d/volumes" Nov 02 19:54:53.048016 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:53.047975 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq"] Nov 02 19:54:53.091022 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:53.090990 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f421424a-fqpqq"] Nov 02 19:54:54.190056 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:54:54.190001 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ab6e1b-42b4-4b21-9012-4dbd1be85462" path="/var/lib/kubelet/pods/d0ab6e1b-42b4-4b21-9012-4dbd1be85462/volumes" Nov 02 19:55:14.756508 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:55:14.756469 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:55:17.040892 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.040848 4966 scope.go:122] "RemoveContainer" containerID="1bcadf5972446e8e9ebb94105d558677d52ba58cc16f36b3b5adec4f34046b71" Nov 02 19:55:17.096520 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.093910 4966 scope.go:122] "RemoveContainer" containerID="2c7fc69641c46d44563c29e724c3c878468db97ddcfe24e687a0cb6b642dc6b2" Nov 02 19:55:17.115036 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.114995 4966 scope.go:122] "RemoveContainer" containerID="1a1ff91ae136c3c0ea0eb5534737c5342a0bcc0880f8d1f727664a6fbfd8fed1" Nov 02 19:55:17.127465 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.127434 4966 scope.go:122] "RemoveContainer" containerID="83f94295d7a9f1538954ebe0947ca4471b9ee5ad2be4e5d1308f312d07ad3627" Nov 02 19:55:17.138923 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.138820 4966 scope.go:122] "RemoveContainer" containerID="b0433db0a66eb1ed538a021ed4b1e8573a41884baa562732949466ca07b1efb5" Nov 02 19:55:17.153547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.153465 4966 scope.go:122] "RemoveContainer" containerID="8ad5d4d8dffc106419cd5cc6b02a38771bc7b57a333d21ede248de4b66bd0564" Nov 02 19:55:17.171789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.171756 4966 scope.go:122] "RemoveContainer" containerID="e9f75a606924babdac73154df40b7dc96bd8775da1b406178273766df2500ed9" Nov 02 19:55:17.182267 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.182247 4966 scope.go:122] "RemoveContainer" containerID="c8cfb9ad2debeecdbde675a137b420827e04d46c8777748918373421c1a547b7" Nov 02 19:55:17.195468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.195447 4966 scope.go:122] "RemoveContainer" containerID="bb5ca403e9d5ef9783fc422e75b83d62a79a01d3ace5db9e9143354e9cbc3621" Nov 02 19:55:17.209547 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.209520 4966 scope.go:122] "RemoveContainer" containerID="617ce219a29f3265e5411530c0e2aa328f5b1802ec5b36ccc9c325ab103a0c52" Nov 02 19:55:17.220259 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.220235 4966 scope.go:122] "RemoveContainer" containerID="b77206b6a3a0f5578085c029224677fc463aa62c7c334b800280a2210daf457d" Nov 02 19:55:17.236205 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.236182 4966 scope.go:122] "RemoveContainer" containerID="a443f0c04c9fb2483206d4dda11b2cb5de7740597c7c79aed6b6d98e059086b9" Nov 02 19:55:17.247146 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.247126 4966 scope.go:122] "RemoveContainer" containerID="b40946ec6d6437ef7fee647a904edc10169c68a6a73f9c3bdd367f6ca47a70f5" Nov 02 19:55:17.257914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.257892 4966 scope.go:122] "RemoveContainer" containerID="714b400a92a926f640f260437b12c5ad13da7aaae409e801f85b0a3f696d4d99" Nov 02 19:55:17.269051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:17.269028 4966 scope.go:122] "RemoveContainer" containerID="8baa1ab625b11e025ab83849d762c1ed5e5819673adb4ccf17e59e57b8641fb6" Nov 02 19:55:21.121719 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:21.121675 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2"] Nov 02 19:55:21.122313 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:21.121899 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" containerID="cri-o://8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106" gracePeriod=30 Nov 02 19:55:21.312930 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:21.312878 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56"] Nov 02 19:55:21.313175 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:21.313149 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" containerID="cri-o://bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f" gracePeriod=30 Nov 02 19:55:21.319264 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:21.319234 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8"] Nov 02 19:55:21.319535 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:21.319492 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" containerID="cri-o://b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813" gracePeriod=30 Nov 02 19:55:22.538058 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.538017 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8"] Nov 02 19:55:22.539809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.539778 4966 cpu_manager.go:414] "containerMap: removing container" podUID="96100327-a10a-4fcf-a9e0-e7457db3a986" containerName="revoke-keys" Nov 02 19:55:22.539970 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.539956 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96100327-a10a-4fcf-a9e0-e7457db3a986" containerName="revoke-keys" Nov 02 19:55:22.540106 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.540095 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" Nov 02 19:55:22.540189 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.540178 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" Nov 02 19:55:22.540339 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.540325 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" Nov 02 19:55:22.540464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.540450 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" Nov 02 19:55:22.540782 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.540763 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" Nov 02 19:55:22.540908 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.540897 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="96100327-a10a-4fcf-a9e0-e7457db3a986" containerName="revoke-keys" Nov 02 19:55:22.560402 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.560359 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb3fa1b-d973-4e78-8340-a204313c171f" containerName="istio-proxy" Nov 02 19:55:22.596321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.593580 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8"] Nov 02 19:55:22.596321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.593735 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.735077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.735035 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:55:22.735371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.735086 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:55:22.735371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.735200 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:55:22.736829 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.736774 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:55:22.773135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.773095 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.773363 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.773147 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.773442 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.773410 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5t5\" (UniqueName: \"kubernetes.io/projected/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-kube-api-access-mb5t5\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.875370 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.875249 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.875560 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.875367 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.875560 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.875516 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5t5\" (UniqueName: \"kubernetes.io/projected/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-kube-api-access-mb5t5\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.876183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.876153 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.878406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.877980 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.928460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.928425 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5t5\" (UniqueName: \"kubernetes.io/projected/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-kube-api-access-mb5t5\") pod \"maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:22.979211 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:22.979176 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:23.366602 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:55:23.366569 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a52df3_ccfe_4e73_b873_7e8b5cf14a29.slice/crio-c9032dd808ef2ebc05ae412170a4111044a3a4a47cd4736500b27858cfc945cf WatchSource:0}: Error finding container c9032dd808ef2ebc05ae412170a4111044a3a4a47cd4736500b27858cfc945cf: Status 404 returned error can't find the container with id c9032dd808ef2ebc05ae412170a4111044a3a4a47cd4736500b27858cfc945cf Nov 02 19:55:23.366746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:23.366721 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8"] Nov 02 19:55:23.694810 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:23.694718 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" event={"ID":"18a52df3-ccfe-4e73-b873-7e8b5cf14a29","Type":"ContainerStarted","Data":"c9032dd808ef2ebc05ae412170a4111044a3a4a47cd4736500b27858cfc945cf"} Nov 02 19:55:24.767591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:24.767542 4966 generic.go:356] "Generic (PLEG): container finished" podID="18a52df3-ccfe-4e73-b873-7e8b5cf14a29" containerID="dffc9b568839e325383c9bd19fc2fe6aa8969933661d44ab03d77bc4a0bc319e" exitCode=0 Nov 02 19:55:24.768406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:24.768364 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" event={"ID":"18a52df3-ccfe-4e73-b873-7e8b5cf14a29","Type":"ContainerDied","Data":"dffc9b568839e325383c9bd19fc2fe6aa8969933661d44ab03d77bc4a0bc319e"} Nov 02 19:55:26.034113 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.034085 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:26.172370 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.172337 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-kube-api-access-mb5t5\" (UniqueName: \"kubernetes.io/projected/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-kube-api-access-mb5t5\") pod \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " Nov 02 19:55:26.172502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.172400 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-xks-ca\") pod \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " Nov 02 19:55:26.172502 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.172431 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-openshift-ca\") pod \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\" (UID: \"18a52df3-ccfe-4e73-b873-7e8b5cf14a29\") " Nov 02 19:55:26.172866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.172836 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-openshift-ca" pod "18a52df3-ccfe-4e73-b873-7e8b5cf14a29" (UID: "18a52df3-ccfe-4e73-b873-7e8b5cf14a29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:26.173213 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.173190 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.174675 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.174648 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-kube-api-access-mb5t5" pod "18a52df3-ccfe-4e73-b873-7e8b5cf14a29" (UID: "18a52df3-ccfe-4e73-b873-7e8b5cf14a29"). InnerVolumeSpecName "kube-api-access-mb5t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.176452 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.176406 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-xks-ca" pod "18a52df3-ccfe-4e73-b873-7e8b5cf14a29" (UID: "18a52df3-ccfe-4e73-b873-7e8b5cf14a29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:26.274438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.274405 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb5t5\" (UniqueName: \"kubernetes.io/projected/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-kube-api-access-mb5t5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.274438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.274435 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a52df3-ccfe-4e73-b873-7e8b5cf14a29-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.455102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.455076 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:55:26.578935 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.578901 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-podinfo\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.578996 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-certs\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579036 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-envoy\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579240 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-kube-api-access-kqx2t\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-kube-api-access-kqx2t\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579419 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579312 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-certs" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.579587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579452 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-token\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-token\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579690 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579653 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-data\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579770 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579710 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a4e90c-b1b7-4414-9d14-9355e04a0511-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a4e90c-b1b7-4414-9d14-9355e04a0511-istiod-ca-cert\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579871 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579771 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-credential-socket\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.579871 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.579807 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-socket\") pod \"01a4e90c-b1b7-4414-9d14-9355e04a0511\" (UID: \"01a4e90c-b1b7-4414-9d14-9355e04a0511\") " Nov 02 19:55:26.580663 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.580623 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.580772 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.580695 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-credential-socket" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.580891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.580804 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-socket" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.580891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.580863 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a4e90c-b1b7-4414-9d14-9355e04a0511-istiod-ca-cert" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:26.581020 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.580942 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-data" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.583490 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.583451 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-token" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.590797 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.590765 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-kube-api-access-kqx2t" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "kube-api-access-kqx2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.590919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.590887 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-envoy" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.592512 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.592484 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-podinfo" pod "01a4e90c-b1b7-4414-9d14-9355e04a0511" (UID: "01a4e90c-b1b7-4414-9d14-9355e04a0511"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:26.605628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.605586 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:55:26.680978 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.680953 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:55:26.681126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681029 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-socket\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681071 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-kube-api-access-qwvt9\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-kube-api-access-qwvt9\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681093 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istiod-ca-cert\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681292 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681189 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-data\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681292 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681233 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-token\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-token\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681292 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681276 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-credential-socket\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681365 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-podinfo\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681380 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-socket" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.681475 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681415 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-certs\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681490 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-data" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.681628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681541 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-envoy\") pod \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\" (UID: \"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b\") " Nov 02 19:55:26.681737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681643 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istiod-ca-cert" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:26.681801 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681739 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-certs" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.681901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.681879 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-credential-socket" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.682374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682352 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682373 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682382 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682392 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a4e90c-b1b7-4414-9d14-9355e04a0511-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682400 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682408 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682417 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682425 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682433 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682442 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682450 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a4e90c-b1b7-4414-9d14-9355e04a0511-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682458 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.682556 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.682467 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqx2t\" (UniqueName: \"kubernetes.io/projected/01a4e90c-b1b7-4414-9d14-9355e04a0511-kube-api-access-kqx2t\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.683524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.683497 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-token" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.683751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.683728 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-envoy" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.683855 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.683830 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-podinfo" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:26.683855 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.683845 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-kube-api-access-qwvt9" pod "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" (UID: "7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b"). InnerVolumeSpecName "kube-api-access-qwvt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.783222 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783187 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-envoy\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783238 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istiod-ca-cert\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783342 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-data\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783415 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-kube-api-access-grqwf\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-kube-api-access-grqwf\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783623 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783498 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-token\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-token\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783698 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783682 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-podinfo\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783704 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-data" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.783754 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783720 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-certs\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783744 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istiod-ca-cert" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:26.783858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783836 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-credential-socket\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.783964 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.783892 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-socket\") pod \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\" (UID: \"a75f0bed-989b-437c-a99f-85eb67f5cb0c\") " Nov 02 19:55:26.784240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784215 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-certs" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.784422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784399 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-socket" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784729 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784757 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784772 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwvt9\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-kube-api-access-qwvt9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784786 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784799 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784799 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-credential-socket" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784814 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784848 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.784920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.784875 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.786128 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.786067 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-kube-api-access-grqwf" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "kube-api-access-grqwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.786257 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.786145 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-envoy" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:55:26.786995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.786972 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-token" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:26.787376 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.787359 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-podinfo" pod "a75f0bed-989b-437c-a99f-85eb67f5cb0c" (UID: "a75f0bed-989b-437c-a99f-85eb67f5cb0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:55:26.798563 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.798540 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" Nov 02 19:55:26.798563 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.798556 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8" event={"ID":"18a52df3-ccfe-4e73-b873-7e8b5cf14a29","Type":"ContainerDied","Data":"c9032dd808ef2ebc05ae412170a4111044a3a4a47cd4736500b27858cfc945cf"} Nov 02 19:55:26.798760 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.798597 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9032dd808ef2ebc05ae412170a4111044a3a4a47cd4736500b27858cfc945cf" Nov 02 19:55:26.800133 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.800113 4966 generic.go:356] "Generic (PLEG): container finished" podID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerID="b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813" exitCode=0 Nov 02 19:55:26.800248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.800160 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" Nov 02 19:55:26.800248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.800197 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" event={"ID":"a75f0bed-989b-437c-a99f-85eb67f5cb0c","Type":"ContainerDied","Data":"b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813"} Nov 02 19:55:26.800416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.800249 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8" event={"ID":"a75f0bed-989b-437c-a99f-85eb67f5cb0c","Type":"ContainerDied","Data":"dac25b0eee7cc3aa041f1fe8adef1726f9c6f68786b02606599058d1b6b6585f"} Nov 02 19:55:26.800416 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.800280 4966 scope.go:122] "RemoveContainer" containerID="b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813" Nov 02 19:55:26.802229 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.802199 4966 generic.go:356] "Generic (PLEG): container finished" podID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerID="8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106" exitCode=0 Nov 02 19:55:26.802369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.802238 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" Nov 02 19:55:26.802369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.802272 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" event={"ID":"01a4e90c-b1b7-4414-9d14-9355e04a0511","Type":"ContainerDied","Data":"8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106"} Nov 02 19:55:26.802369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.802347 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2" event={"ID":"01a4e90c-b1b7-4414-9d14-9355e04a0511","Type":"ContainerDied","Data":"7a8a0f46dc07e7dfec45ff6c923eff79200db19669b639622a6bf53258cf3d7f"} Nov 02 19:55:26.804001 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.803977 4966 generic.go:356] "Generic (PLEG): container finished" podID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerID="bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f" exitCode=0 Nov 02 19:55:26.804112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.804021 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" Nov 02 19:55:26.804112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.804055 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" event={"ID":"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b","Type":"ContainerDied","Data":"bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f"} Nov 02 19:55:26.804112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.804106 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56" event={"ID":"7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b","Type":"ContainerDied","Data":"f0814f44085ba7dc694e61bd6ae4fcbcb08b045d5fde82ef115d2ac783d9e151"} Nov 02 19:55:26.813604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.813569 4966 scope.go:122] "RemoveContainer" containerID="b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813" Nov 02 19:55:26.813861 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:55:26.813843 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813\": container with ID starting with b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813 not found: ID does not exist" containerID="b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813" Nov 02 19:55:26.813955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.813872 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813"} err="failed to get container status \"b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813\": rpc error: code = NotFound desc = could not find container \"b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813\": container with ID starting with b590cf23da9fb5623cf052e25ad23fdbfa69259f6f109911776ea4be3a6ed813 not found: ID does not exist" Nov 02 19:55:26.813955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.813897 4966 scope.go:122] "RemoveContainer" containerID="8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106" Nov 02 19:55:26.827570 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.827536 4966 scope.go:122] "RemoveContainer" containerID="8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106" Nov 02 19:55:26.827951 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:55:26.827927 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106\": container with ID starting with 8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106 not found: ID does not exist" containerID="8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106" Nov 02 19:55:26.828072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.827956 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106"} err="failed to get container status \"8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106\": rpc error: code = NotFound desc = could not find container \"8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106\": container with ID starting with 8b8a1c9b3b732e7caedf128e8acc063c89684c6636a70c7e4a94cb52ddc5d106 not found: ID does not exist" Nov 02 19:55:26.828072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.827975 4966 scope.go:122] "RemoveContainer" containerID="bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f" Nov 02 19:55:26.841806 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.841778 4966 scope.go:122] "RemoveContainer" containerID="bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f" Nov 02 19:55:26.842164 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:55:26.842127 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f\": container with ID starting with bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f not found: ID does not exist" containerID="bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f" Nov 02 19:55:26.842265 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.842166 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f"} err="failed to get container status \"bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f\": rpc error: code = NotFound desc = could not find container \"bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f\": container with ID starting with bec30fd37035fe9597a16a38c26c73d641703fdc8ce9179731a0dd6e3453ba5f not found: ID does not exist" Nov 02 19:55:26.885854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.885820 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.885854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.885850 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqwf\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-kube-api-access-grqwf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.885854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.885860 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.886128 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.885869 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75f0bed-989b-437c-a99f-85eb67f5cb0c-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.886128 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.885878 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75f0bed-989b-437c-a99f-85eb67f5cb0c-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:26.984499 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.984457 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56"] Nov 02 19:55:26.990075 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:26.990050 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-k4p56"] Nov 02 19:55:27.091948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:27.091913 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2"] Nov 02 19:55:27.099574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:27.099546 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-gscm2"] Nov 02 19:55:27.206115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:27.206075 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8"] Nov 02 19:55:27.206115 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:27.206126 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-8qpc8"] Nov 02 19:55:28.184496 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:28.184463 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" path="/var/lib/kubelet/pods/01a4e90c-b1b7-4414-9d14-9355e04a0511/volumes" Nov 02 19:55:28.185005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:28.184989 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" path="/var/lib/kubelet/pods/7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b/volumes" Nov 02 19:55:28.185535 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:28.185514 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" path="/var/lib/kubelet/pods/a75f0bed-989b-437c-a99f-85eb67f5cb0c/volumes" Nov 02 19:55:44.754755 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:55:44.754711 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:55:51.521246 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.521208 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7"] Nov 02 19:55:51.522217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522199 4966 cpu_manager.go:414] "containerMap: removing container" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" Nov 02 19:55:51.522275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522220 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" Nov 02 19:55:51.522275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522256 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" Nov 02 19:55:51.522275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522264 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" Nov 02 19:55:51.522408 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522287 4966 cpu_manager.go:414] "containerMap: removing container" podUID="18a52df3-ccfe-4e73-b873-7e8b5cf14a29" containerName="revoke-keys" Nov 02 19:55:51.522408 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522310 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a52df3-ccfe-4e73-b873-7e8b5cf14a29" containerName="revoke-keys" Nov 02 19:55:51.522408 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522337 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" Nov 02 19:55:51.522408 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522345 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" Nov 02 19:55:51.522526 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522498 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a52df3-ccfe-4e73-b873-7e8b5cf14a29" containerName="revoke-keys" Nov 02 19:55:51.522526 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522515 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75f0bed-989b-437c-a99f-85eb67f5cb0c" containerName="istio-proxy" Nov 02 19:55:51.522590 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522530 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a4e90c-b1b7-4414-9d14-9355e04a0511" containerName="istio-proxy" Nov 02 19:55:51.522590 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.522544 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2bf8e9-8ac7-4b1b-b057-68cb0b57a69b" containerName="istio-proxy" Nov 02 19:55:51.527323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.527288 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.577606 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.577565 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7"] Nov 02 19:55:51.619345 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.618081 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:55:51.619345 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.618216 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:55:51.619345 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.618282 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:55:51.621277 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.621240 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:55:51.682183 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.682146 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda04212-4462-46d2-b818-5afac0270838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.682390 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.682359 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda04212-4462-46d2-b818-5afac0270838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.682454 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.682415 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xcx\" (UniqueName: \"kubernetes.io/projected/dda04212-4462-46d2-b818-5afac0270838-kube-api-access-85xcx\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.784870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.784762 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda04212-4462-46d2-b818-5afac0270838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.785031 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.784974 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda04212-4462-46d2-b818-5afac0270838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.785081 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.785044 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85xcx\" (UniqueName: \"kubernetes.io/projected/dda04212-4462-46d2-b818-5afac0270838-kube-api-access-85xcx\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.785866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.785841 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda04212-4462-46d2-b818-5afac0270838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.787674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.787651 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda04212-4462-46d2-b818-5afac0270838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.840127 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.840090 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xcx\" (UniqueName: \"kubernetes.io/projected/dda04212-4462-46d2-b818-5afac0270838-kube-api-access-85xcx\") pod \"maas-api-revoke-sub-63b88623-f4nw7\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:51.842703 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:51.842664 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:52.223048 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:55:52.223006 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda04212_4462_46d2_b818_5afac0270838.slice/crio-90ff30424e57724e780f17229282cb2d75532a80230cadebbc9bcfb44998c8d5 WatchSource:0}: Error finding container 90ff30424e57724e780f17229282cb2d75532a80230cadebbc9bcfb44998c8d5: Status 404 returned error can't find the container with id 90ff30424e57724e780f17229282cb2d75532a80230cadebbc9bcfb44998c8d5 Nov 02 19:55:52.223171 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:52.223148 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7"] Nov 02 19:55:53.056777 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:53.056735 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" event={"ID":"dda04212-4462-46d2-b818-5afac0270838","Type":"ContainerStarted","Data":"ca4b693cfb6cdd0c6a319624df2b56730d101d102ae6ff1ddeddb835e43205d5"} Nov 02 19:55:53.057251 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:53.056819 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" event={"ID":"dda04212-4462-46d2-b818-5afac0270838","Type":"ContainerStarted","Data":"90ff30424e57724e780f17229282cb2d75532a80230cadebbc9bcfb44998c8d5"} Nov 02 19:55:53.162233 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:53.162178 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" podStartSLOduration=2.162162972 podStartE2EDuration="2.162162972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:53.160477989 +0000 UTC m=+1659.109133828" watchObservedRunningTime="2026-11-02 19:55:53.162162972 +0000 UTC m=+1659.110818811" Nov 02 19:55:54.068423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:54.068385 4966 generic.go:356] "Generic (PLEG): container finished" podID="dda04212-4462-46d2-b818-5afac0270838" containerID="ca4b693cfb6cdd0c6a319624df2b56730d101d102ae6ff1ddeddb835e43205d5" exitCode=0 Nov 02 19:55:54.068902 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:54.068472 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" event={"ID":"dda04212-4462-46d2-b818-5afac0270838","Type":"ContainerDied","Data":"ca4b693cfb6cdd0c6a319624df2b56730d101d102ae6ff1ddeddb835e43205d5"} Nov 02 19:55:55.273862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.273831 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:55.438976 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.438879 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda04212-4462-46d2-b818-5afac0270838-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda04212-4462-46d2-b818-5afac0270838-maas-api-xks-ca\") pod \"dda04212-4462-46d2-b818-5afac0270838\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " Nov 02 19:55:55.439151 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.438991 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda04212-4462-46d2-b818-5afac0270838-kube-api-access-85xcx\" (UniqueName: \"kubernetes.io/projected/dda04212-4462-46d2-b818-5afac0270838-kube-api-access-85xcx\") pod \"dda04212-4462-46d2-b818-5afac0270838\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " Nov 02 19:55:55.439151 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.439024 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda04212-4462-46d2-b818-5afac0270838-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda04212-4462-46d2-b818-5afac0270838-maas-api-openshift-ca\") pod \"dda04212-4462-46d2-b818-5afac0270838\" (UID: \"dda04212-4462-46d2-b818-5afac0270838\") " Nov 02 19:55:55.439914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.439891 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda04212-4462-46d2-b818-5afac0270838-maas-api-openshift-ca" pod "dda04212-4462-46d2-b818-5afac0270838" (UID: "dda04212-4462-46d2-b818-5afac0270838"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:55:55.443125 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.443083 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda04212-4462-46d2-b818-5afac0270838-kube-api-access-85xcx" pod "dda04212-4462-46d2-b818-5afac0270838" (UID: "dda04212-4462-46d2-b818-5afac0270838"). InnerVolumeSpecName "kube-api-access-85xcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:55.443258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.443090 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda04212-4462-46d2-b818-5afac0270838-maas-api-xks-ca" pod "dda04212-4462-46d2-b818-5afac0270838" (UID: "dda04212-4462-46d2-b818-5afac0270838"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:55:55.551322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.545644 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda04212-4462-46d2-b818-5afac0270838-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:55.551322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.545693 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85xcx\" (UniqueName: \"kubernetes.io/projected/dda04212-4462-46d2-b818-5afac0270838-kube-api-access-85xcx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:55.551322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:55.545711 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda04212-4462-46d2-b818-5afac0270838-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:56.088463 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:56.088427 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" Nov 02 19:55:56.088696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:56.088459 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7" event={"ID":"dda04212-4462-46d2-b818-5afac0270838","Type":"ContainerDied","Data":"90ff30424e57724e780f17229282cb2d75532a80230cadebbc9bcfb44998c8d5"} Nov 02 19:55:56.088696 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:55:56.088507 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ff30424e57724e780f17229282cb2d75532a80230cadebbc9bcfb44998c8d5" Nov 02 19:56:06.519939 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.519887 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz"] Nov 02 19:56:06.520563 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.520169 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" containerID="cri-o://cd4633fcca89dc7eaf1c0fb7d1dcf4083734f289510d5f1921e5afa5a39f3a61" gracePeriod=30 Nov 02 19:56:06.548406 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.548368 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth"] Nov 02 19:56:06.548634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.548605 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" containerID="cri-o://62fdb37fb509f64f6b1ee701ebbc7e3244a83d9f519a7a39a4582306f61908fb" gracePeriod=30 Nov 02 19:56:06.553209 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.553178 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj"] Nov 02 19:56:06.553449 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.553424 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" containerID="cri-o://5a6742d41faa265bd077d802dde3ffc7a97e9ecbc0abd03902e0f192b145a3ca" gracePeriod=30 Nov 02 19:56:06.594858 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.594820 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx"] Nov 02 19:56:06.595079 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:06.595028 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" containerID="cri-o://ad6df2f2ee40418ce7b16e1c524773595f9834bbb2565af0c2496d315411171e" gracePeriod=30 Nov 02 19:56:08.620031 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:08.619990 4966 patch_prober.go:29] interesting pod/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:56:08.620626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:08.620061 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:56:10.143010 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:10.142973 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8"] Nov 02 19:56:10.187870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:10.187840 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4099cbd3-q4kw8"] Nov 02 19:56:11.820601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.818213 4966 generic.go:356] "Generic (PLEG): container finished" podID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerID="ad6df2f2ee40418ce7b16e1c524773595f9834bbb2565af0c2496d315411171e" exitCode=0 Nov 02 19:56:11.820601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.818345 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" event={"ID":"aa43ea8b-4430-49d2-940a-dcd655dbc83e","Type":"ContainerDied","Data":"ad6df2f2ee40418ce7b16e1c524773595f9834bbb2565af0c2496d315411171e"} Nov 02 19:56:11.820601 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.818399 4966 scope.go:122] "RemoveContainer" containerID="e93c9d49d20fd7570dd8464a758368a4f40284ae2ff48912c2740c8d840ed7b9" Nov 02 19:56:11.824048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.823251 4966 generic.go:356] "Generic (PLEG): container finished" podID="b5792127-6974-484d-8ca2-8cb8962664ef" containerID="62fdb37fb509f64f6b1ee701ebbc7e3244a83d9f519a7a39a4582306f61908fb" exitCode=0 Nov 02 19:56:11.824048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.823461 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" event={"ID":"b5792127-6974-484d-8ca2-8cb8962664ef","Type":"ContainerDied","Data":"62fdb37fb509f64f6b1ee701ebbc7e3244a83d9f519a7a39a4582306f61908fb"} Nov 02 19:56:11.826225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.825832 4966 generic.go:356] "Generic (PLEG): container finished" podID="e7b4d281-badc-4056-8530-1117729fde32" containerID="5a6742d41faa265bd077d802dde3ffc7a97e9ecbc0abd03902e0f192b145a3ca" exitCode=0 Nov 02 19:56:11.826225 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.826060 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" event={"ID":"e7b4d281-badc-4056-8530-1117729fde32","Type":"ContainerDied","Data":"5a6742d41faa265bd077d802dde3ffc7a97e9ecbc0abd03902e0f192b145a3ca"} Nov 02 19:56:11.829147 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.828315 4966 generic.go:356] "Generic (PLEG): container finished" podID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerID="cd4633fcca89dc7eaf1c0fb7d1dcf4083734f289510d5f1921e5afa5a39f3a61" exitCode=0 Nov 02 19:56:11.829147 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.828421 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" event={"ID":"62aa0383-eacb-4759-9e8d-9535b20289c7","Type":"ContainerDied","Data":"cd4633fcca89dc7eaf1c0fb7d1dcf4083734f289510d5f1921e5afa5a39f3a61"} Nov 02 19:56:11.950554 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:11.950518 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:56:12.022033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.022005 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:56:12.035445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035418 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-token\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-token\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.035613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035469 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-envoy\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.035613 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035591 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-podinfo\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.035736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035623 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-credential-socket\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.035736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035678 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istiod-ca-cert\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.035736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035717 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-kube-api-access-4lp88\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-kube-api-access-4lp88\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035754 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-envoy\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035806 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-certs\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035865 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-data\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035898 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-credential-socket\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.035975 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62aa0383-eacb-4759-9e8d-9535b20289c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62aa0383-eacb-4759-9e8d-9535b20289c7-istiod-ca-cert\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036016 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-data\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036019 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-credential-socket" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036058 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-socket\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036100 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-socket\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036210 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-podinfo\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036260 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-certs\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036290 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-kube-api-access-4kzxm\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-kube-api-access-4kzxm\") pod \"62aa0383-eacb-4759-9e8d-9535b20289c7\" (UID: \"62aa0383-eacb-4759-9e8d-9535b20289c7\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036337 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-token\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-token\") pod \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\" (UID: \"aa43ea8b-4430-49d2-940a-dcd655dbc83e\") " Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036890 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-data" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.037129 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.037152 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.037373 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-certs" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.037559 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-data" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.037689 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-credential-socket" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.037890 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62aa0383-eacb-4759-9e8d-9535b20289c7-istiod-ca-cert" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.036890 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istiod-ca-cert" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.038190 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-socket" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.038455 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-certs" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.038608 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.038522 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-socket" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.041445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.041286 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-token" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.041445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.041286 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-envoy" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.041445 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.041378 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-podinfo" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:56:12.041626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.041523 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-token" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.043058 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.043014 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-kube-api-access-4lp88" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "kube-api-access-4lp88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.043243 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.043215 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-podinfo" pod "aa43ea8b-4430-49d2-940a-dcd655dbc83e" (UID: "aa43ea8b-4430-49d2-940a-dcd655dbc83e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:56:12.043421 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.043393 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-kube-api-access-4kzxm" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "kube-api-access-4kzxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.043876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.043859 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-envoy" pod "62aa0383-eacb-4759-9e8d-9535b20289c7" (UID: "62aa0383-eacb-4759-9e8d-9535b20289c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.044112 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.044089 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:56:12.047143 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.047124 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:56:12.138530 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138454 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-socket\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.138530 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138494 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-certs\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.138735 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138578 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-istio-token\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-istio-token\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.138735 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138650 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-data\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.138825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138757 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-data\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.138825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138780 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-socket" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.138825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138791 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-certs\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.138944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138832 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-credential-socket\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.138944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138834 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-certs" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.138944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138872 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b4d281-badc-4056-8530-1117729fde32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b4d281-badc-4056-8530-1117729fde32-istiod-ca-cert\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.138944 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138903 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-envoy\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.139163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138941 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-data" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.139163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.138988 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5792127-6974-484d-8ca2-8cb8962664ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5792127-6974-484d-8ca2-8cb8962664ef-istio-podinfo\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.139163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139026 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-data" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.139163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139039 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-credential-socket\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.139163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139087 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7b4d281-badc-4056-8530-1117729fde32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b4d281-badc-4056-8530-1117729fde32-istio-podinfo\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.139163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139118 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-certs" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.139163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139129 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-socket\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139188 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-istio-token\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-istio-token\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139233 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-kube-api-access-l6tdn\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-kube-api-access-l6tdn\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139270 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-credential-socket" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139278 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b4d281-badc-4056-8530-1117729fde32-istiod-ca-cert" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139314 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-envoy\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139391 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-kube-api-access-bvpg9\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-kube-api-access-bvpg9\") pod \"e7b4d281-badc-4056-8530-1117729fde32\" (UID: \"e7b4d281-badc-4056-8530-1117729fde32\") " Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139415 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-credential-socket" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139423 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5792127-6974-484d-8ca2-8cb8962664ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5792127-6974-484d-8ca2-8cb8962664ef-istiod-ca-cert\") pod \"b5792127-6974-484d-8ca2-8cb8962664ef\" (UID: \"b5792127-6974-484d-8ca2-8cb8962664ef\") " Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139606 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-socket" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.139700 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5792127-6974-484d-8ca2-8cb8962664ef-istiod-ca-cert" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140508 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140536 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140553 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140568 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140587 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140601 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7b4d281-badc-4056-8530-1117729fde32-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140615 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140629 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140645 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140660 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lp88\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-kube-api-access-4lp88\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140675 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140690 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140704 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140718 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140731 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140747 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62aa0383-eacb-4759-9e8d-9535b20289c7-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140762 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5792127-6974-484d-8ca2-8cb8962664ef-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140776 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140791 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62aa0383-eacb-4759-9e8d-9535b20289c7-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140807 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140822 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140837 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62aa0383-eacb-4759-9e8d-9535b20289c7-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140851 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa43ea8b-4430-49d2-940a-dcd655dbc83e-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140866 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kzxm\" (UniqueName: \"kubernetes.io/projected/62aa0383-eacb-4759-9e8d-9535b20289c7-kube-api-access-4kzxm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140880 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa43ea8b-4430-49d2-940a-dcd655dbc83e-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.140894 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.141877 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-envoy" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.141881 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-envoy" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:12.143591 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.143057 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-kube-api-access-l6tdn" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "kube-api-access-l6tdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.145108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.143811 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-istio-token" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.145108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.144070 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-kube-api-access-bvpg9" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "kube-api-access-bvpg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.145108 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.144243 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5792127-6974-484d-8ca2-8cb8962664ef-istio-podinfo" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:56:12.145495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.145471 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7b4d281-badc-4056-8530-1117729fde32-istio-podinfo" pod "e7b4d281-badc-4056-8530-1117729fde32" (UID: "e7b4d281-badc-4056-8530-1117729fde32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:56:12.146742 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.146719 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-istio-token" pod "b5792127-6974-484d-8ca2-8cb8962664ef" (UID: "b5792127-6974-484d-8ca2-8cb8962664ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:12.192087 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.192048 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a52df3-ccfe-4e73-b873-7e8b5cf14a29" path="/var/lib/kubelet/pods/18a52df3-ccfe-4e73-b873-7e8b5cf14a29/volumes" Nov 02 19:56:12.242609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242576 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.242609 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242615 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6tdn\" (UniqueName: \"kubernetes.io/projected/b5792127-6974-484d-8ca2-8cb8962664ef-kube-api-access-l6tdn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.242794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242633 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7b4d281-badc-4056-8530-1117729fde32-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.242794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242649 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvpg9\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-kube-api-access-bvpg9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.242794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242663 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7b4d281-badc-4056-8530-1117729fde32-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.242794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242678 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5792127-6974-484d-8ca2-8cb8962664ef-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.242794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242704 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5792127-6974-484d-8ca2-8cb8962664ef-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.242794 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.242726 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7b4d281-badc-4056-8530-1117729fde32-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:12.838825 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.838723 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" event={"ID":"e7b4d281-badc-4056-8530-1117729fde32","Type":"ContainerDied","Data":"96567790ad55d7a63b0f1ab4ba5f9450852d30e63be1f78a41227f78f0942434"} Nov 02 19:56:12.839321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.838844 4966 scope.go:122] "RemoveContainer" containerID="5a6742d41faa265bd077d802dde3ffc7a97e9ecbc0abd03902e0f192b145a3ca" Nov 02 19:56:12.839321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.838957 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj" Nov 02 19:56:12.840746 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.840724 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" Nov 02 19:56:12.840870 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.840796 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz" event={"ID":"62aa0383-eacb-4759-9e8d-9535b20289c7","Type":"ContainerDied","Data":"bb4de03f78e36a96d8c51dd9cde77528201b5c0a0b628831a0899acbc76fbd3d"} Nov 02 19:56:12.843248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.843224 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" Nov 02 19:56:12.843866 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.843841 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx" event={"ID":"aa43ea8b-4430-49d2-940a-dcd655dbc83e","Type":"ContainerDied","Data":"4d1cdd4e831fdf638d71870cebb06f0c061ee89c99d32fe2f138ac38db85c5ba"} Nov 02 19:56:12.845605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.845584 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" event={"ID":"b5792127-6974-484d-8ca2-8cb8962664ef","Type":"ContainerDied","Data":"69e3e4ca640749347209615a87a725a64735078b58d6ab5da115e9b6c5ba2627"} Nov 02 19:56:12.845705 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.845624 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth" Nov 02 19:56:12.855107 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.855086 4966 scope.go:122] "RemoveContainer" containerID="cd4633fcca89dc7eaf1c0fb7d1dcf4083734f289510d5f1921e5afa5a39f3a61" Nov 02 19:56:12.868026 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.868000 4966 scope.go:122] "RemoveContainer" containerID="ad6df2f2ee40418ce7b16e1c524773595f9834bbb2565af0c2496d315411171e" Nov 02 19:56:12.880067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.880041 4966 scope.go:122] "RemoveContainer" containerID="62fdb37fb509f64f6b1ee701ebbc7e3244a83d9f519a7a39a4582306f61908fb" Nov 02 19:56:12.996163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:12.996128 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj"] Nov 02 19:56:13.042344 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:13.042312 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qfwrj"] Nov 02 19:56:13.140192 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:13.140096 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx"] Nov 02 19:56:13.145422 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:13.145391 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-2t6jx"] Nov 02 19:56:13.242393 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:13.242351 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz"] Nov 02 19:56:13.246426 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:13.246392 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-qf9bz"] Nov 02 19:56:13.347067 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:13.347032 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth"] Nov 02 19:56:13.350899 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:13.350872 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26cfbe-openshift-default-69d49cc7f5-nvzth"] Nov 02 19:56:14.063655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.063615 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78"] Nov 02 19:56:14.065102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065047 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" Nov 02 19:56:14.065102 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065102 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065131 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065139 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065164 4966 cpu_manager.go:414] "containerMap: removing container" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065171 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065185 4966 cpu_manager.go:414] "containerMap: removing container" podUID="dda04212-4462-46d2-b818-5afac0270838" containerName="revoke-keys" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065192 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda04212-4462-46d2-b818-5afac0270838" containerName="revoke-keys" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065230 4966 cpu_manager.go:414] "containerMap: removing container" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065237 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065253 4966 cpu_manager.go:414] "containerMap: removing container" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" Nov 02 19:56:14.065312 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065261 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" Nov 02 19:56:14.065995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065423 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" containerName="istio-proxy" Nov 02 19:56:14.065995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065446 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b4d281-badc-4056-8530-1117729fde32" containerName="istio-proxy" Nov 02 19:56:14.065995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065458 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" Nov 02 19:56:14.065995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065467 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda04212-4462-46d2-b818-5afac0270838" containerName="revoke-keys" Nov 02 19:56:14.065995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065482 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" containerName="istio-proxy" Nov 02 19:56:14.065995 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.065523 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" containerName="istio-proxy" Nov 02 19:56:14.072521 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.072456 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.120752 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.120722 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-89a22a\"" Nov 02 19:56:14.120955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.120832 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-89a22a-dockercfg-zznml\"" Nov 02 19:56:14.125133 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.125107 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78"] Nov 02 19:56:14.178615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.173901 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdh5m\" (UniqueName: \"kubernetes.io/projected/722d1f95-7b32-43ea-9007-0230a25eaca4-kube-api-access-bdh5m\") pod \"payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.178615 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.174196 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722d1f95-7b32-43ea-9007-0230a25eaca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.288352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.284372 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722d1f95-7b32-43ea-9007-0230a25eaca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.288352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.284483 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdh5m\" (UniqueName: \"kubernetes.io/projected/722d1f95-7b32-43ea-9007-0230a25eaca4-kube-api-access-bdh5m\") pod \"payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.288352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.285708 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722d1f95-7b32-43ea-9007-0230a25eaca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.307495 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.306382 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62aa0383-eacb-4759-9e8d-9535b20289c7" path="/var/lib/kubelet/pods/62aa0383-eacb-4759-9e8d-9535b20289c7/volumes" Nov 02 19:56:14.312024 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.308921 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa43ea8b-4430-49d2-940a-dcd655dbc83e" path="/var/lib/kubelet/pods/aa43ea8b-4430-49d2-940a-dcd655dbc83e/volumes" Nov 02 19:56:14.312024 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.309684 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5792127-6974-484d-8ca2-8cb8962664ef" path="/var/lib/kubelet/pods/b5792127-6974-484d-8ca2-8cb8962664ef/volumes" Nov 02 19:56:14.327616 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.324522 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b4d281-badc-4056-8530-1117729fde32" path="/var/lib/kubelet/pods/e7b4d281-badc-4056-8530-1117729fde32/volumes" Nov 02 19:56:14.327616 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.325022 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6"] Nov 02 19:56:14.368683 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.366613 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdh5m\" (UniqueName: \"kubernetes.io/projected/722d1f95-7b32-43ea-9007-0230a25eaca4-kube-api-access-bdh5m\") pod \"payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.382522 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.382487 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.402391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.402353 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcgpf\" (UniqueName: \"kubernetes.io/projected/7144b232-f9ed-4b70-979e-70e066b021ad-kube-api-access-lcgpf\") pod \"payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.402563 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.402543 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7144b232-f9ed-4b70-979e-70e066b021ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.426093 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.426061 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6"] Nov 02 19:56:14.444258 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.444232 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-89a22a-dockercfg-zznml\"" Nov 02 19:56:14.453527 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.453499 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:14.504725 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.504680 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcgpf\" (UniqueName: \"kubernetes.io/projected/7144b232-f9ed-4b70-979e-70e066b021ad-kube-api-access-lcgpf\") pod \"payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.504887 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.504870 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7144b232-f9ed-4b70-979e-70e066b021ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.505641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.505614 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7144b232-f9ed-4b70-979e-70e066b021ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.560053 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.559989 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcgpf\" (UniqueName: \"kubernetes.io/projected/7144b232-f9ed-4b70-979e-70e066b021ad-kube-api-access-lcgpf\") pod \"payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.713948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.713856 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:14.756116 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:56:14.756081 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:56:14.836347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.836271 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78"] Nov 02 19:56:14.878277 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:14.878243 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" event={"ID":"722d1f95-7b32-43ea-9007-0230a25eaca4","Type":"ContainerStarted","Data":"5a4810cf11fd09c4512e58354e763f6a5e6556ea2e5e5804bc27cfae080c0365"} Nov 02 19:56:15.093468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:15.093438 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6"] Nov 02 19:56:15.094077 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:56:15.094052 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7144b232_f9ed_4b70_979e_70e066b021ad.slice/crio-b3d6f797e9b9ff18d95a0e0c101d212a04c5694664355dd546815a9d2a0a56ea WatchSource:0}: Error finding container b3d6f797e9b9ff18d95a0e0c101d212a04c5694664355dd546815a9d2a0a56ea: Status 404 returned error can't find the container with id b3d6f797e9b9ff18d95a0e0c101d212a04c5694664355dd546815a9d2a0a56ea Nov 02 19:56:15.888357 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:15.888319 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" event={"ID":"7144b232-f9ed-4b70-979e-70e066b021ad","Type":"ContainerStarted","Data":"1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542"} Nov 02 19:56:15.888632 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:15.888383 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" event={"ID":"7144b232-f9ed-4b70-979e-70e066b021ad","Type":"ContainerStarted","Data":"b3d6f797e9b9ff18d95a0e0c101d212a04c5694664355dd546815a9d2a0a56ea"} Nov 02 19:56:15.890167 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:15.890132 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" event={"ID":"722d1f95-7b32-43ea-9007-0230a25eaca4","Type":"ContainerStarted","Data":"940269dda5b94f911beeade3369d5927026de291f912da591da4470b946e28d6"} Nov 02 19:56:15.890289 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:15.890192 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:15.997394 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:15.997335 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" podStartSLOduration=1.997318426 podStartE2EDuration="1.997318426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:15.992713002 +0000 UTC m=+1681.941368841" watchObservedRunningTime="2026-11-02 19:56:15.997318426 +0000 UTC m=+1681.945974261" Nov 02 19:56:16.100928 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:16.100873 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" podStartSLOduration=2.100857766 podStartE2EDuration="2.100857766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:16.096680914 +0000 UTC m=+1682.045336754" watchObservedRunningTime="2026-11-02 19:56:16.100857766 +0000 UTC m=+1682.049513642" Nov 02 19:56:16.904806 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:16.904737 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:17.642634 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:17.642595 4966 scope.go:122] "RemoveContainer" containerID="e6f025d44b24493e374eb0ca33f25c74920953b2a9897ae47e67286e220a6d5c" Nov 02 19:56:17.656423 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:17.656396 4966 scope.go:122] "RemoveContainer" containerID="75d04b7d7d1a212e7daf4c926755141aed1015e88c8a013292f6ac47aa056a3c" Nov 02 19:56:17.668918 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:17.668895 4966 scope.go:122] "RemoveContainer" containerID="df6ed0bb07c45587a056026ed352427846bed541c4a1fda335f393a9b2d7f0fd" Nov 02 19:56:17.681568 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:17.681545 4966 scope.go:122] "RemoveContainer" containerID="f6ce41b77117689521165eaf6869162b01b1b78b39f1a5f9192968142a60412a" Nov 02 19:56:17.693430 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:17.693410 4966 scope.go:122] "RemoveContainer" containerID="5fe993b0ab84a6c428deec7481b5fe277c565bda0461f9f933af25038f282356" Nov 02 19:56:17.709934 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:17.709911 4966 scope.go:122] "RemoveContainer" containerID="904b5a6a54c1afa3a51f6cfc821f260e1687034947d17d2bb23826bcd5722b49" Nov 02 19:56:17.722787 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:17.722755 4966 scope.go:122] "RemoveContainer" containerID="a52743a0936882429dcc7add2b5f8508bc18b95163e53b1b04cf30639ff104ea" Nov 02 19:56:25.859245 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:25.859209 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq"] Nov 02 19:56:25.865763 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:25.865735 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:25.916617 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:25.916573 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq"] Nov 02 19:56:25.957231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:25.957174 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:56:25.957231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:25.957175 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:56:25.957231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:25.957208 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:56:25.957509 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:25.957263 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:56:26.028511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.026632 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.028511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.026767 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226wb\" (UniqueName: \"kubernetes.io/projected/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-kube-api-access-226wb\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.028511 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.026858 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.130367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.128447 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226wb\" (UniqueName: \"kubernetes.io/projected/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-kube-api-access-226wb\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.130367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.128542 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.130367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.128706 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.130367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.129573 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.132340 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.131832 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.185641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.185012 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226wb\" (UniqueName: \"kubernetes.io/projected/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-kube-api-access-226wb\") pod \"maas-api-revoke-sub-830941e5-ftvvq\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.185641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.185256 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:26.421457 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.421374 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs"] Nov 02 19:56:26.440751 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.435136 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.497323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.489800 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b0cc24-dockercfg-27dtr\"" Nov 02 19:56:26.497323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.490026 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b0cc24\"" Nov 02 19:56:26.511042 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.511002 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs"] Nov 02 19:56:26.566319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.566263 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f6e754-f50e-4df5-b53b-995aeb4cad39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.587367 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.566346 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cshj\" (UniqueName: \"kubernetes.io/projected/e9f6e754-f50e-4df5-b53b-995aeb4cad39-kube-api-access-9cshj\") pod \"payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.615638 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.611983 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf"] Nov 02 19:56:26.623322 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.620718 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:26.671336 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.671279 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f6e754-f50e-4df5-b53b-995aeb4cad39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.671576 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.671367 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cshj\" (UniqueName: \"kubernetes.io/projected/e9f6e754-f50e-4df5-b53b-995aeb4cad39-kube-api-access-9cshj\") pod \"payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.673371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.673320 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f6e754-f50e-4df5-b53b-995aeb4cad39-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.674337 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.674309 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf"] Nov 02 19:56:26.752668 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.752272 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cshj\" (UniqueName: \"kubernetes.io/projected/e9f6e754-f50e-4df5-b53b-995aeb4cad39-kube-api-access-9cshj\") pod \"payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.761741 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.761707 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:26.781663 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.781606 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsc9b\" (UniqueName: \"kubernetes.io/projected/c4751352-2544-4c5f-b8ef-9f85e50c899f-kube-api-access-tsc9b\") pod \"payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:26.782060 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.781837 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4751352-2544-4c5f-b8ef-9f85e50c899f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:26.826903 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.826869 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq"] Nov 02 19:56:26.827167 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:56:26.827142 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e9f0f3_d6a0_421f_9384_d7f0d89e1b3c.slice/crio-c5221105713992d8f8c2cc8180b61a9ce364287592f88f9a04bb6cf228948f66 WatchSource:0}: Error finding container c5221105713992d8f8c2cc8180b61a9ce364287592f88f9a04bb6cf228948f66: Status 404 returned error can't find the container with id c5221105713992d8f8c2cc8180b61a9ce364287592f88f9a04bb6cf228948f66 Nov 02 19:56:26.884005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.882991 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsc9b\" (UniqueName: \"kubernetes.io/projected/c4751352-2544-4c5f-b8ef-9f85e50c899f-kube-api-access-tsc9b\") pod \"payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:26.884005 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.883210 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4751352-2544-4c5f-b8ef-9f85e50c899f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:26.884674 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.884200 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4751352-2544-4c5f-b8ef-9f85e50c899f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:26.908272 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.908087 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 19:56:26.937946 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.937589 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsc9b\" (UniqueName: \"kubernetes.io/projected/c4751352-2544-4c5f-b8ef-9f85e50c899f-kube-api-access-tsc9b\") pod \"payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:26.955409 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:26.955387 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:27.053447 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:27.053347 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" event={"ID":"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c","Type":"ContainerStarted","Data":"c5221105713992d8f8c2cc8180b61a9ce364287592f88f9a04bb6cf228948f66"} Nov 02 19:56:27.170498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:27.170449 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs"] Nov 02 19:56:27.175654 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:56:27.175621 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f6e754_f50e_4df5_b53b_995aeb4cad39.slice/crio-b15dea58bc7fa36ed7ded4359d8a28635a4b5aec61a7d94d26062736139e27d3 WatchSource:0}: Error finding container b15dea58bc7fa36ed7ded4359d8a28635a4b5aec61a7d94d26062736139e27d3: Status 404 returned error can't find the container with id b15dea58bc7fa36ed7ded4359d8a28635a4b5aec61a7d94d26062736139e27d3 Nov 02 19:56:27.351507 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:27.351469 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf"] Nov 02 19:56:27.352592 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:56:27.352563 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4751352_2544_4c5f_b8ef_9f85e50c899f.slice/crio-86f60049e41b42526b698545ba294caa375a103cd8953ea48d5ef19fa453d804 WatchSource:0}: Error finding container 86f60049e41b42526b698545ba294caa375a103cd8953ea48d5ef19fa453d804: Status 404 returned error can't find the container with id 86f60049e41b42526b698545ba294caa375a103cd8953ea48d5ef19fa453d804 Nov 02 19:56:27.915587 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:27.915555 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 19:56:28.073790 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.073746 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" event={"ID":"c4751352-2544-4c5f-b8ef-9f85e50c899f","Type":"ContainerStarted","Data":"ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f"} Nov 02 19:56:28.080487 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.080406 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" event={"ID":"c4751352-2544-4c5f-b8ef-9f85e50c899f","Type":"ContainerStarted","Data":"86f60049e41b42526b698545ba294caa375a103cd8953ea48d5ef19fa453d804"} Nov 02 19:56:28.080778 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.080758 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:28.080877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.080803 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:28.080877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.080829 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" event={"ID":"e9f6e754-f50e-4df5-b53b-995aeb4cad39","Type":"ContainerStarted","Data":"c85a34867184ccfa12ac026c6ad4546406435598ba7fe9235bb10b171ca98000"} Nov 02 19:56:28.080877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.080851 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" event={"ID":"e9f6e754-f50e-4df5-b53b-995aeb4cad39","Type":"ContainerStarted","Data":"b15dea58bc7fa36ed7ded4359d8a28635a4b5aec61a7d94d26062736139e27d3"} Nov 02 19:56:28.081955 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.081929 4966 generic.go:356] "Generic (PLEG): container finished" podID="86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" containerID="ec3664489993a01f6c23f630b2630d4d4dec6e599439625cbfc5033683602b98" exitCode=0 Nov 02 19:56:28.082068 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.082051 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" event={"ID":"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c","Type":"ContainerDied","Data":"ec3664489993a01f6c23f630b2630d4d4dec6e599439625cbfc5033683602b98"} Nov 02 19:56:28.181743 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.181686 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" podStartSLOduration=2.181666254 podStartE2EDuration="2.181666254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:28.178254929 +0000 UTC m=+1694.126910766" watchObservedRunningTime="2026-11-02 19:56:28.181666254 +0000 UTC m=+1694.130322094" Nov 02 19:56:28.288372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:28.287261 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" podStartSLOduration=2.287245238 podStartE2EDuration="2.287245238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:28.28242145 +0000 UTC m=+1694.231077302" watchObservedRunningTime="2026-11-02 19:56:28.287245238 +0000 UTC m=+1694.235901077" Nov 02 19:56:29.522321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.518371 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:29.604957 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.604912 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-xks-ca\") pod \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " Nov 02 19:56:29.605178 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.605047 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-kube-api-access-226wb\" (UniqueName: \"kubernetes.io/projected/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-kube-api-access-226wb\") pod \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " Nov 02 19:56:29.605369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.605346 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-openshift-ca\") pod \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\" (UID: \"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c\") " Nov 02 19:56:29.605790 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.605750 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-openshift-ca" pod "86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" (UID: "86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:56:29.606062 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.606032 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:29.607408 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.607380 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-xks-ca" pod "86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" (UID: "86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:56:29.608991 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.608967 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-kube-api-access-226wb" pod "86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" (UID: "86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c"). InnerVolumeSpecName "kube-api-access-226wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:29.721643 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.721611 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:29.721886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:29.721864 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226wb\" (UniqueName: \"kubernetes.io/projected/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c-kube-api-access-226wb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:30.115395 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:30.112008 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" Nov 02 19:56:30.115718 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:30.115688 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq" event={"ID":"86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c","Type":"ContainerDied","Data":"c5221105713992d8f8c2cc8180b61a9ce364287592f88f9a04bb6cf228948f66"} Nov 02 19:56:30.115846 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:30.115784 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5221105713992d8f8c2cc8180b61a9ce364287592f88f9a04bb6cf228948f66" Nov 02 19:56:34.050446 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:34.050407 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66"] Nov 02 19:56:34.093630 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:34.093595 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cb1d10-rdp66"] Nov 02 19:56:34.184478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:34.184447 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae" path="/var/lib/kubelet/pods/2ef0a458-1093-4de3-8c1c-8ab62d0fd8ae/volumes" Nov 02 19:56:39.100269 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:39.100235 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 19:56:40.114989 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:40.114957 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 19:56:44.754493 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:56:44.754455 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:56:51.675868 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.675825 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg"] Nov 02 19:56:51.677605 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.677575 4966 cpu_manager.go:414] "containerMap: removing container" podUID="86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" containerName="revoke-keys" Nov 02 19:56:51.677789 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.677767 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" containerName="revoke-keys" Nov 02 19:56:51.678169 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.678140 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" containerName="revoke-keys" Nov 02 19:56:51.689332 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.684061 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:51.785320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.785262 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd584daa-3872-4b5b-884c-d69457bdf6ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:51.785927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.785902 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48grq\" (UniqueName: \"kubernetes.io/projected/cd584daa-3872-4b5b-884c-d69457bdf6ab-kube-api-access-48grq\") pod \"payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:51.860606 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.860571 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg"] Nov 02 19:56:51.863196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.863169 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-24ef8b-dockercfg-2htqc\"" Nov 02 19:56:51.864070 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.863477 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-24ef8b\"" Nov 02 19:56:51.868033 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.868011 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n"] Nov 02 19:56:51.872508 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.872492 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:51.891771 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.891501 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48grq\" (UniqueName: \"kubernetes.io/projected/cd584daa-3872-4b5b-884c-d69457bdf6ab-kube-api-access-48grq\") pod \"payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:51.891771 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.891557 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rgr\" (UniqueName: \"kubernetes.io/projected/ed90b751-2446-4d32-9e72-80255277cd17-kube-api-access-q8rgr\") pod \"payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:51.891771 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.891653 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd584daa-3872-4b5b-884c-d69457bdf6ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:51.891771 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.891753 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed90b751-2446-4d32-9e72-80255277cd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:51.893366 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.893339 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd584daa-3872-4b5b-884c-d69457bdf6ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:51.910877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.910843 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n"] Nov 02 19:56:51.945629 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.945561 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48grq\" (UniqueName: \"kubernetes.io/projected/cd584daa-3872-4b5b-884c-d69457bdf6ab-kube-api-access-48grq\") pod \"payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:51.993434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.993398 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rgr\" (UniqueName: \"kubernetes.io/projected/ed90b751-2446-4d32-9e72-80255277cd17-kube-api-access-q8rgr\") pod \"payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:51.993626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.993532 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed90b751-2446-4d32-9e72-80255277cd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:51.994347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:51.994292 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed90b751-2446-4d32-9e72-80255277cd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:52.029499 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.029465 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:52.046279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.046254 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rgr\" (UniqueName: \"kubernetes.io/projected/ed90b751-2446-4d32-9e72-80255277cd17-kube-api-access-q8rgr\") pod \"payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:52.204346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.204226 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:52.409346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.409314 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg"] Nov 02 19:56:52.409460 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:56:52.409399 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd584daa_3872_4b5b_884c_d69457bdf6ab.slice/crio-b0643c86b9181218dd806edfcbd8226dddae4c9f5309b4586b7f914e9c986869 WatchSource:0}: Error finding container b0643c86b9181218dd806edfcbd8226dddae4c9f5309b4586b7f914e9c986869: Status 404 returned error can't find the container with id b0643c86b9181218dd806edfcbd8226dddae4c9f5309b4586b7f914e9c986869 Nov 02 19:56:52.529832 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.529798 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" event={"ID":"cd584daa-3872-4b5b-884c-d69457bdf6ab","Type":"ContainerStarted","Data":"315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22"} Nov 02 19:56:52.530043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.529858 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" event={"ID":"cd584daa-3872-4b5b-884c-d69457bdf6ab","Type":"ContainerStarted","Data":"b0643c86b9181218dd806edfcbd8226dddae4c9f5309b4586b7f914e9c986869"} Nov 02 19:56:52.530043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.529884 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:56:52.594170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.594139 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n"] Nov 02 19:56:52.594462 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:56:52.594439 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded90b751_2446_4d32_9e72_80255277cd17.slice/crio-c60ac48e0758c46098b49501ab4bae39ab3e154f6c4f1b6a0ae370b7a4480dac WatchSource:0}: Error finding container c60ac48e0758c46098b49501ab4bae39ab3e154f6c4f1b6a0ae370b7a4480dac: Status 404 returned error can't find the container with id c60ac48e0758c46098b49501ab4bae39ab3e154f6c4f1b6a0ae370b7a4480dac Nov 02 19:56:52.635681 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:52.635631 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" podStartSLOduration=1.635616956 podStartE2EDuration="1.635616956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:52.634847726 +0000 UTC m=+1718.583503565" watchObservedRunningTime="2026-11-02 19:56:52.635616956 +0000 UTC m=+1718.584272795" Nov 02 19:56:53.539702 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:53.539658 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" event={"ID":"ed90b751-2446-4d32-9e72-80255277cd17","Type":"ContainerStarted","Data":"e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745"} Nov 02 19:56:53.540149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:53.539729 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" event={"ID":"ed90b751-2446-4d32-9e72-80255277cd17","Type":"ContainerStarted","Data":"c60ac48e0758c46098b49501ab4bae39ab3e154f6c4f1b6a0ae370b7a4480dac"} Nov 02 19:56:53.540149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:53.540040 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:56:53.648173 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:56:53.647999 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" podStartSLOduration=2.647981062 podStartE2EDuration="2.647981062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:53.646579771 +0000 UTC m=+1719.595235610" watchObservedRunningTime="2026-11-02 19:56:53.647981062 +0000 UTC m=+1719.596636902" Nov 02 19:57:03.543941 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:03.543910 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 19:57:04.555549 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:04.555516 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 19:57:14.753731 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:57:14.753693 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:57:17.881464 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:17.881427 4966 scope.go:122] "RemoveContainer" containerID="19594b231c3d78655cede0a7c83bdacbb5f40b3f701fd85fb11ca7d3f8116b95" Nov 02 19:57:37.302689 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:37.302654 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f"] Nov 02 19:57:37.303320 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:37.302905 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" containerID="cri-o://0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf" gracePeriod=30 Nov 02 19:57:37.351539 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:37.351496 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs"] Nov 02 19:57:37.351809 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:37.351775 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" containerID="cri-o://832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a" gracePeriod=30 Nov 02 19:57:40.213534 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:40.213496 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:57:40.213994 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:40.213553 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:57:42.438278 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:57:42.438243 4966 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267809e1_83dd_445f_8646_2a77e3b9bb97.slice/crio-conmon-832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a.scope\": RecentStats: unable to find data in memory cache]" Nov 02 19:57:42.646126 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.646089 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:57:42.721099 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.721067 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:57:42.727323 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727270 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-certs\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727325 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-istio-token\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-istio-token\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727440 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727360 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-envoy\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727463 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-socket\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727533 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727510 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12faba46-6929-4454-9353-d310edbf42a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12faba46-6929-4454-9353-d310edbf42a9-istio-podinfo\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727630 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727539 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-certs" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.727630 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727604 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-kube-api-access-n242t\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-kube-api-access-n242t\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727697 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-data\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727740 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-socket" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.727796 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727771 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12faba46-6929-4454-9353-d310edbf42a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12faba46-6929-4454-9353-d310edbf42a9-istiod-ca-cert\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727914 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727832 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-credential-socket\") pod \"12faba46-6929-4454-9353-d310edbf42a9\" (UID: \"12faba46-6929-4454-9353-d310edbf42a9\") " Nov 02 19:57:42.727985 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.727963 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-data" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.728210 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.728187 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-credential-socket" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.728346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.728242 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12faba46-6929-4454-9353-d310edbf42a9-istiod-ca-cert" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:57:42.728590 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.728565 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.728680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.728597 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.728680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.728612 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.728680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.728627 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.728680 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.728640 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12faba46-6929-4454-9353-d310edbf42a9-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.729776 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.729743 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-envoy" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.729889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.729824 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-kube-api-access-n242t" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "kube-api-access-n242t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:42.729889 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.729870 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-istio-token" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:42.730069 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.730052 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12faba46-6929-4454-9353-d310edbf42a9-istio-podinfo" pod "12faba46-6929-4454-9353-d310edbf42a9" (UID: "12faba46-6929-4454-9353-d310edbf42a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:57:42.829708 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.829676 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267809e1-83dd-445f-8646-2a77e3b9bb97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267809e1-83dd-445f-8646-2a77e3b9bb97-istiod-ca-cert\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.829921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.829723 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-data\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.829921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.829865 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-envoy\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.829921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.829911 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-podinfo\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.830109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.829950 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-socket\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.830109 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830024 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-certs\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.830276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830149 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-data" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.830276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830184 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-token\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-token\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.830276 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830252 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267809e1-83dd-445f-8646-2a77e3b9bb97-istiod-ca-cert" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:57:42.830460 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830261 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-credential-socket\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.830517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830350 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-kube-api-access-rvqrn\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-kube-api-access-rvqrn\") pod \"267809e1-83dd-445f-8646-2a77e3b9bb97\" (UID: \"267809e1-83dd-445f-8646-2a77e3b9bb97\") " Nov 02 19:57:42.830517 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830490 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-credential-socket" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.830693 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830671 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-socket" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.830758 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830690 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-certs" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.831011 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.830991 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831011 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831010 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12faba46-6929-4454-9353-d310edbf42a9-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831020 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12faba46-6929-4454-9353-d310edbf42a9-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831030 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831039 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n242t\" (UniqueName: \"kubernetes.io/projected/12faba46-6929-4454-9353-d310edbf42a9-kube-api-access-n242t\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831048 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831058 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831066 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267809e1-83dd-445f-8646-2a77e3b9bb97-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.831158 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.831074 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.833170 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.833141 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-kube-api-access-rvqrn" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "kube-api-access-rvqrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:42.833434 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.833407 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-envoy" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:42.833566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.833477 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-podinfo" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 19:57:42.833641 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.833630 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-token" pod "267809e1-83dd-445f-8646-2a77e3b9bb97" (UID: "267809e1-83dd-445f-8646-2a77e3b9bb97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:42.932954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.932863 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.932954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.932907 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvqrn\" (UniqueName: \"kubernetes.io/projected/267809e1-83dd-445f-8646-2a77e3b9bb97-kube-api-access-rvqrn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.932954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.932924 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:42.932954 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:42.932940 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267809e1-83dd-445f-8646-2a77e3b9bb97-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:43.100246 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.100210 4966 generic.go:356] "Generic (PLEG): container finished" podID="12faba46-6929-4454-9353-d310edbf42a9" containerID="0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf" exitCode=0 Nov 02 19:57:43.100471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.100271 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" Nov 02 19:57:43.100471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.100291 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" event={"ID":"12faba46-6929-4454-9353-d310edbf42a9","Type":"ContainerDied","Data":"0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf"} Nov 02 19:57:43.100471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.100399 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f" event={"ID":"12faba46-6929-4454-9353-d310edbf42a9","Type":"ContainerDied","Data":"11adebac78f9b66610deaef25b5141039d07b783a96df71c51f8a778e635ec9f"} Nov 02 19:57:43.100471 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.100430 4966 scope.go:122] "RemoveContainer" containerID="0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf" Nov 02 19:57:43.101920 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.101894 4966 generic.go:356] "Generic (PLEG): container finished" podID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerID="832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a" exitCode=0 Nov 02 19:57:43.102043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.101970 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" Nov 02 19:57:43.102043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.101975 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" event={"ID":"267809e1-83dd-445f-8646-2a77e3b9bb97","Type":"ContainerDied","Data":"832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a"} Nov 02 19:57:43.102043 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.102037 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs" event={"ID":"267809e1-83dd-445f-8646-2a77e3b9bb97","Type":"ContainerDied","Data":"06d5e192c92722a8afd2f000d38741c0abed588c0adcdbb30c418849cb0b8b4b"} Nov 02 19:57:43.113530 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.113515 4966 scope.go:122] "RemoveContainer" containerID="0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf" Nov 02 19:57:43.113812 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:57:43.113794 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf\": container with ID starting with 0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf not found: ID does not exist" containerID="0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf" Nov 02 19:57:43.113862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.113822 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf"} err="failed to get container status \"0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf\": rpc error: code = NotFound desc = could not find container \"0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf\": container with ID starting with 0fce9acff00efd50cfa7e3bdfa2f14c52529534adf2e34ad16fa89d8a7f1c0bf not found: ID does not exist" Nov 02 19:57:43.113862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.113839 4966 scope.go:122] "RemoveContainer" containerID="832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a" Nov 02 19:57:43.123656 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.123637 4966 scope.go:122] "RemoveContainer" containerID="832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a" Nov 02 19:57:43.123931 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:57:43.123912 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a\": container with ID starting with 832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a not found: ID does not exist" containerID="832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a" Nov 02 19:57:43.123992 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.123942 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a"} err="failed to get container status \"832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a\": rpc error: code = NotFound desc = could not find container \"832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a\": container with ID starting with 832e45c70c4a2ea01a17cccffc8eb90adb70646d0dfcc14f6d479f201e2e128a not found: ID does not exist" Nov 02 19:57:43.255886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.255850 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f"] Nov 02 19:57:43.259919 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.259888 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-9h87f"] Nov 02 19:57:43.360384 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.360344 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs"] Nov 02 19:57:43.364248 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:43.364214 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-fqqbs"] Nov 02 19:57:44.185349 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:44.185281 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12faba46-6929-4454-9353-d310edbf42a9" path="/var/lib/kubelet/pods/12faba46-6929-4454-9353-d310edbf42a9/volumes" Nov 02 19:57:44.185971 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:57:44.185953 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" path="/var/lib/kubelet/pods/267809e1-83dd-445f-8646-2a77e3b9bb97/volumes" Nov 02 19:57:44.770990 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:57:44.770919 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:58:07.281854 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.281812 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn"] Nov 02 19:58:07.282804 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.282782 4966 cpu_manager.go:414] "containerMap: removing container" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" Nov 02 19:58:07.282863 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.282806 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" Nov 02 19:58:07.282863 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.282850 4966 cpu_manager.go:414] "containerMap: removing container" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" Nov 02 19:58:07.282863 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.282860 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" Nov 02 19:58:07.283040 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.283028 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="12faba46-6929-4454-9353-d310edbf42a9" containerName="istio-proxy" Nov 02 19:58:07.283077 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.283050 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="267809e1-83dd-445f-8646-2a77e3b9bb97" containerName="istio-proxy" Nov 02 19:58:07.287283 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.287261 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.335819 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.335787 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:58:07.339346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.339316 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn"] Nov 02 19:58:07.369082 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.369048 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.369283 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.369257 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vfr\" (UniqueName: \"kubernetes.io/projected/0bd23151-254b-4588-ab5f-03d658975eb3-kube-api-access-v6vfr\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.369385 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.369330 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.382700 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.382673 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 19:58:07.383671 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.383650 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:58:07.383953 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.383938 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:58:07.470646 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.470601 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vfr\" (UniqueName: \"kubernetes.io/projected/0bd23151-254b-4588-ab5f-03d658975eb3-kube-api-access-v6vfr\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.470837 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.470679 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.470837 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.470793 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.471604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.471577 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.473270 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.473249 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.523387 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.523352 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vfr\" (UniqueName: \"kubernetes.io/projected/0bd23151-254b-4588-ab5f-03d658975eb3-kube-api-access-v6vfr\") pod \"maas-api-revoke-sub-9e365778-hggqn\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.601697 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.601621 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:07.985319 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:07.985279 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn"] Nov 02 19:58:07.986631 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:58:07.986599 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd23151_254b_4588_ab5f_03d658975eb3.slice/crio-5ce8b941f0eb74206fad2f4812cec79e7e4dc5a727fc2d8863ade8d3ca336d38 WatchSource:0}: Error finding container 5ce8b941f0eb74206fad2f4812cec79e7e4dc5a727fc2d8863ade8d3ca336d38: Status 404 returned error can't find the container with id 5ce8b941f0eb74206fad2f4812cec79e7e4dc5a727fc2d8863ade8d3ca336d38 Nov 02 19:58:08.350628 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:08.350582 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" event={"ID":"0bd23151-254b-4588-ab5f-03d658975eb3","Type":"ContainerStarted","Data":"5ce8b941f0eb74206fad2f4812cec79e7e4dc5a727fc2d8863ade8d3ca336d38"} Nov 02 19:58:09.362347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:09.362231 4966 generic.go:356] "Generic (PLEG): container finished" podID="0bd23151-254b-4588-ab5f-03d658975eb3" containerID="ab1198d675b62c9800749d5afdd2782b6c39ad3127c1c2d1d15d4bc24130f686" exitCode=0 Nov 02 19:58:09.362347 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:09.362273 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" event={"ID":"0bd23151-254b-4588-ab5f-03d658975eb3","Type":"ContainerDied","Data":"ab1198d675b62c9800749d5afdd2782b6c39ad3127c1c2d1d15d4bc24130f686"} Nov 02 19:58:10.549779 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.549739 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:10.718498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.718467 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-openshift-ca\") pod \"0bd23151-254b-4588-ab5f-03d658975eb3\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " Nov 02 19:58:10.718655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.718533 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-xks-ca\") pod \"0bd23151-254b-4588-ab5f-03d658975eb3\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " Nov 02 19:58:10.718655 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.718598 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd23151-254b-4588-ab5f-03d658975eb3-kube-api-access-v6vfr\" (UniqueName: \"kubernetes.io/projected/0bd23151-254b-4588-ab5f-03d658975eb3-kube-api-access-v6vfr\") pod \"0bd23151-254b-4588-ab5f-03d658975eb3\" (UID: \"0bd23151-254b-4588-ab5f-03d658975eb3\") " Nov 02 19:58:10.718924 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.718895 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-openshift-ca" pod "0bd23151-254b-4588-ab5f-03d658975eb3" (UID: "0bd23151-254b-4588-ab5f-03d658975eb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:58:10.719196 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.719173 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:58:10.720805 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.720784 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-xks-ca" pod "0bd23151-254b-4588-ab5f-03d658975eb3" (UID: "0bd23151-254b-4588-ab5f-03d658975eb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:58:10.720929 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.720910 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd23151-254b-4588-ab5f-03d658975eb3-kube-api-access-v6vfr" pod "0bd23151-254b-4588-ab5f-03d658975eb3" (UID: "0bd23151-254b-4588-ab5f-03d658975eb3"). InnerVolumeSpecName "kube-api-access-v6vfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:58:10.820163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.820131 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd23151-254b-4588-ab5f-03d658975eb3-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:58:10.820163 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:10.820158 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6vfr\" (UniqueName: \"kubernetes.io/projected/0bd23151-254b-4588-ab5f-03d658975eb3-kube-api-access-v6vfr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:58:11.384592 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:11.384562 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" Nov 02 19:58:11.384799 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:11.384596 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn" event={"ID":"0bd23151-254b-4588-ab5f-03d658975eb3","Type":"ContainerDied","Data":"5ce8b941f0eb74206fad2f4812cec79e7e4dc5a727fc2d8863ade8d3ca336d38"} Nov 02 19:58:11.384799 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:11.384652 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ce8b941f0eb74206fad2f4812cec79e7e4dc5a727fc2d8863ade8d3ca336d38" Nov 02 19:58:14.753580 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:58:14.753510 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:58:14.788069 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:14.788036 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:58:14.804927 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:14.804896 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:58:14.840950 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:14.840908 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 19:58:14.854386 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:14.854359 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 19:58:17.915219 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:17.915178 4966 scope.go:122] "RemoveContainer" containerID="3112dcb98464d46c5e15f443aa24594faa76c84c1524108588538954e34dae2e" Nov 02 19:58:17.928879 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:17.928841 4966 scope.go:122] "RemoveContainer" containerID="e37f78aa5be137dbaa34ae0203b7889c977ff6c8b96c17f6644cf4f23fa1aa63" Nov 02 19:58:17.941394 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:17.941365 4966 scope.go:122] "RemoveContainer" containerID="a4ab87284a3b562fdb55f483cab5ac6c853e5322d8fbac165a00a55a19e36a5a" Nov 02 19:58:42.968886 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:42.968845 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl"] Nov 02 19:58:42.971727 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:42.969806 4966 cpu_manager.go:414] "containerMap: removing container" podUID="0bd23151-254b-4588-ab5f-03d658975eb3" containerName="revoke-keys" Nov 02 19:58:42.971727 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:42.969824 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd23151-254b-4588-ab5f-03d658975eb3" containerName="revoke-keys" Nov 02 19:58:42.971727 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:42.970045 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd23151-254b-4588-ab5f-03d658975eb3" containerName="revoke-keys" Nov 02 19:58:42.974361 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:42.974334 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.024978 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.024935 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl"] Nov 02 19:58:43.033401 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.033364 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa48ba08-be58-4f1c-8834-99b9c46b999f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.033593 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.033411 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ccrj\" (UniqueName: \"kubernetes.io/projected/aa48ba08-be58-4f1c-8834-99b9c46b999f-kube-api-access-8ccrj\") pod \"payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.075369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.074638 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c60a0d-dockercfg-t8sc2\"" Nov 02 19:58:43.075369 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.074751 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c60a0d\"" Nov 02 19:58:43.162389 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.161556 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa48ba08-be58-4f1c-8834-99b9c46b999f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.162389 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.161612 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ccrj\" (UniqueName: \"kubernetes.io/projected/aa48ba08-be58-4f1c-8834-99b9c46b999f-kube-api-access-8ccrj\") pod \"payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.187321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.164920 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa48ba08-be58-4f1c-8834-99b9c46b999f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.188292 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.188251 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l"] Nov 02 19:58:43.199149 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.193993 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.215235 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.215200 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ccrj\" (UniqueName: \"kubernetes.io/projected/aa48ba08-be58-4f1c-8834-99b9c46b999f-kube-api-access-8ccrj\") pod \"payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.231260 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.231192 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l"] Nov 02 19:58:43.263104 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.263064 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgpv\" (UniqueName: \"kubernetes.io/projected/15a0ed26-57d3-46d4-8535-82adda030709-kube-api-access-9kgpv\") pod \"payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.263377 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.263347 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a0ed26-57d3-46d4-8535-82adda030709-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.294960 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.294924 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:43.365193 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.365156 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a0ed26-57d3-46d4-8535-82adda030709-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.365372 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.365312 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgpv\" (UniqueName: \"kubernetes.io/projected/15a0ed26-57d3-46d4-8535-82adda030709-kube-api-access-9kgpv\") pod \"payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.366709 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.366612 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a0ed26-57d3-46d4-8535-82adda030709-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.419089 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.418992 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgpv\" (UniqueName: \"kubernetes.io/projected/15a0ed26-57d3-46d4-8535-82adda030709-kube-api-access-9kgpv\") pod \"payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.513589 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.513486 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:43.702973 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:58:43.702945 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa48ba08_be58_4f1c_8834_99b9c46b999f.slice/crio-7380d8a94fbb27a18b4d053c74fd8171738618412e4a86d63b652c4a691acecc WatchSource:0}: Error finding container 7380d8a94fbb27a18b4d053c74fd8171738618412e4a86d63b652c4a691acecc: Status 404 returned error can't find the container with id 7380d8a94fbb27a18b4d053c74fd8171738618412e4a86d63b652c4a691acecc Nov 02 19:58:43.706282 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.705873 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl"] Nov 02 19:58:43.754428 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.754358 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" event={"ID":"aa48ba08-be58-4f1c-8834-99b9c46b999f","Type":"ContainerStarted","Data":"7380d8a94fbb27a18b4d053c74fd8171738618412e4a86d63b652c4a691acecc"} Nov 02 19:58:43.896113 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:58:43.896079 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a0ed26_57d3_46d4_8535_82adda030709.slice/crio-6435650abfc8f772bedaaf9abbd04769160ca75e86269c6b85c38d6bedd24747 WatchSource:0}: Error finding container 6435650abfc8f772bedaaf9abbd04769160ca75e86269c6b85c38d6bedd24747: Status 404 returned error can't find the container with id 6435650abfc8f772bedaaf9abbd04769160ca75e86269c6b85c38d6bedd24747 Nov 02 19:58:43.897120 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:43.897087 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l"] Nov 02 19:58:44.755076 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:58:44.755015 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:58:44.765209 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:44.765176 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" event={"ID":"15a0ed26-57d3-46d4-8535-82adda030709","Type":"ContainerStarted","Data":"535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0"} Nov 02 19:58:44.765379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:44.765239 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" event={"ID":"15a0ed26-57d3-46d4-8535-82adda030709","Type":"ContainerStarted","Data":"6435650abfc8f772bedaaf9abbd04769160ca75e86269c6b85c38d6bedd24747"} Nov 02 19:58:44.765379 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:44.765271 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:58:44.767255 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:44.767231 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" event={"ID":"aa48ba08-be58-4f1c-8834-99b9c46b999f","Type":"ContainerStarted","Data":"a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76"} Nov 02 19:58:44.767383 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:44.767285 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:44.873862 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:44.873811 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" podStartSLOduration=1.873797264 podStartE2EDuration="1.873797264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:44.869635623 +0000 UTC m=+1830.818291461" watchObservedRunningTime="2026-11-02 19:58:44.873797264 +0000 UTC m=+1830.822453132" Nov 02 19:58:44.979558 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:44.979499 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" podStartSLOduration=2.9794834850000003 podStartE2EDuration="2.979483485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:44.974823831 +0000 UTC m=+1830.923479671" watchObservedRunningTime="2026-11-02 19:58:44.979483485 +0000 UTC m=+1830.928139324" Nov 02 19:58:50.730814 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.730778 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf"] Nov 02 19:58:50.736514 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.736488 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:50.789596 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.789564 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf"] Nov 02 19:58:50.811468 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.811428 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b0606-6b1c-4b6c-903a-1f8532f877a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:50.811673 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.811533 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqljx\" (UniqueName: \"kubernetes.io/projected/af6b0606-6b1c-4b6c-903a-1f8532f877a7-kube-api-access-lqljx\") pod \"payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:50.836352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.835680 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ffdc4d\"" Nov 02 19:58:50.836352 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.835898 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ffdc4d-dockercfg-fwvsv\"" Nov 02 19:58:50.917242 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.917191 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b0606-6b1c-4b6c-903a-1f8532f877a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:50.917544 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.917510 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqljx\" (UniqueName: \"kubernetes.io/projected/af6b0606-6b1c-4b6c-903a-1f8532f877a7-kube-api-access-lqljx\") pod \"payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:50.919640 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.919615 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b0606-6b1c-4b6c-903a-1f8532f877a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:50.923565 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.923530 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s"] Nov 02 19:58:50.934545 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.934515 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:50.992374 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:50.984806 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s"] Nov 02 19:58:51.012279 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.012245 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqljx\" (UniqueName: \"kubernetes.io/projected/af6b0606-6b1c-4b6c-903a-1f8532f877a7-kube-api-access-lqljx\") pod \"payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:51.026811 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.026777 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8cj\" (UniqueName: \"kubernetes.io/projected/2b1d7daa-ab6e-4613-ab73-5497cc565854-kube-api-access-qt8cj\") pod \"payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.026972 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.026934 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1d7daa-ab6e-4613-ab73-5497cc565854-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.055948 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.055915 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:51.128048 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.128006 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8cj\" (UniqueName: \"kubernetes.io/projected/2b1d7daa-ab6e-4613-ab73-5497cc565854-kube-api-access-qt8cj\") pod \"payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.128314 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.128075 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1d7daa-ab6e-4613-ab73-5497cc565854-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.129483 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.129362 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1d7daa-ab6e-4613-ab73-5497cc565854-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.196768 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.196734 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8cj\" (UniqueName: \"kubernetes.io/projected/2b1d7daa-ab6e-4613-ab73-5497cc565854-kube-api-access-qt8cj\") pod \"payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.271877 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.271781 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.474094 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.474055 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf"] Nov 02 19:58:51.474880 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:58:51.474850 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6b0606_6b1c_4b6c_903a_1f8532f877a7.slice/crio-164cff35b2cddf2205c55e15830aeb1ca86b4d7694fff81aae9dfcaddf1939ba WatchSource:0}: Error finding container 164cff35b2cddf2205c55e15830aeb1ca86b4d7694fff81aae9dfcaddf1939ba: Status 404 returned error can't find the container with id 164cff35b2cddf2205c55e15830aeb1ca86b4d7694fff81aae9dfcaddf1939ba Nov 02 19:58:51.649827 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.649796 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s"] Nov 02 19:58:51.652596 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:58:51.652567 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1d7daa_ab6e_4613_ab73_5497cc565854.slice/crio-7933de30d8f799a12c2a9d2e9271273a36c473ffafe2162e61deecc52be954e1 WatchSource:0}: Error finding container 7933de30d8f799a12c2a9d2e9271273a36c473ffafe2162e61deecc52be954e1: Status 404 returned error can't find the container with id 7933de30d8f799a12c2a9d2e9271273a36c473ffafe2162e61deecc52be954e1 Nov 02 19:58:51.832469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.832375 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" event={"ID":"af6b0606-6b1c-4b6c-903a-1f8532f877a7","Type":"ContainerStarted","Data":"18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230"} Nov 02 19:58:51.832469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.832429 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" event={"ID":"af6b0606-6b1c-4b6c-903a-1f8532f877a7","Type":"ContainerStarted","Data":"164cff35b2cddf2205c55e15830aeb1ca86b4d7694fff81aae9dfcaddf1939ba"} Nov 02 19:58:51.832469 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.832460 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:58:51.834364 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.834289 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" event={"ID":"2b1d7daa-ab6e-4613-ab73-5497cc565854","Type":"ContainerStarted","Data":"819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517"} Nov 02 19:58:51.834478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.834389 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" event={"ID":"2b1d7daa-ab6e-4613-ab73-5497cc565854","Type":"ContainerStarted","Data":"7933de30d8f799a12c2a9d2e9271273a36c473ffafe2162e61deecc52be954e1"} Nov 02 19:58:51.834478 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.834414 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:58:51.938433 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:51.938254 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" podStartSLOduration=1.938235858 podStartE2EDuration="1.938235858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:51.937151953 +0000 UTC m=+1837.885807792" watchObservedRunningTime="2026-11-02 19:58:51.938235858 +0000 UTC m=+1837.886891698" Nov 02 19:58:52.046793 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:52.046731 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" podStartSLOduration=2.046713399 podStartE2EDuration="2.046713399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:52.043034009 +0000 UTC m=+1837.991689860" watchObservedRunningTime="2026-11-02 19:58:52.046713399 +0000 UTC m=+1837.995369239" Nov 02 19:58:55.781346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:55.781239 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 19:58:55.781346 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:58:55.781332 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 19:59:02.846756 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:02.846720 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 19:59:02.847202 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:02.846807 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 19:59:09.965776 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:09.965736 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz"] Nov 02 19:59:09.971626 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:09.971598 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.040911 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.040854 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4b0a12\"" Nov 02 19:59:10.044581 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.044547 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz"] Nov 02 19:59:10.081993 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.081954 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4b0a12-dockercfg-f46th\"" Nov 02 19:59:10.113574 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.113542 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft"] Nov 02 19:59:10.118270 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.118246 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.151176 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.151136 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44782bb2-91ba-4c84-bae8-529bb304390a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.151436 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.151354 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v7gk\" (UniqueName: \"kubernetes.io/projected/44782bb2-91ba-4c84-bae8-529bb304390a-kube-api-access-6v7gk\") pod \"payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.176371 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.174148 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft"] Nov 02 19:59:10.253072 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.253039 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44782bb2-91ba-4c84-bae8-529bb304390a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.253275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.253131 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxxpg\" (UniqueName: \"kubernetes.io/projected/c435f75f-ad8b-40fb-b461-fbf070cada9d-kube-api-access-jxxpg\") pod \"payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.253275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.253193 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v7gk\" (UniqueName: \"kubernetes.io/projected/44782bb2-91ba-4c84-bae8-529bb304390a-kube-api-access-6v7gk\") pod \"payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.253275 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.253257 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c435f75f-ad8b-40fb-b461-fbf070cada9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.254038 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.253963 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44782bb2-91ba-4c84-bae8-529bb304390a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.342524 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.342489 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v7gk\" (UniqueName: \"kubernetes.io/projected/44782bb2-91ba-4c84-bae8-529bb304390a-kube-api-access-6v7gk\") pod \"payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.366321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.366013 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxxpg\" (UniqueName: \"kubernetes.io/projected/c435f75f-ad8b-40fb-b461-fbf070cada9d-kube-api-access-jxxpg\") pod \"payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.366321 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.366154 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c435f75f-ad8b-40fb-b461-fbf070cada9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.380316 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.367224 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c435f75f-ad8b-40fb-b461-fbf070cada9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.425324 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.423059 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxxpg\" (UniqueName: \"kubernetes.io/projected/c435f75f-ad8b-40fb-b461-fbf070cada9d-kube-api-access-jxxpg\") pod \"payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.436566 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.436536 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:10.630358 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.630288 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:10.843140 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:10.843106 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft"] Nov 02 19:59:10.843525 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:59:10.843500 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc435f75f_ad8b_40fb_b461_fbf070cada9d.slice/crio-1fad4d1ca0f8132a9d5fd2c5d73ded8378a5834d2f3c3733b66c743d6bc461bb WatchSource:0}: Error finding container 1fad4d1ca0f8132a9d5fd2c5d73ded8378a5834d2f3c3733b66c743d6bc461bb: Status 404 returned error can't find the container with id 1fad4d1ca0f8132a9d5fd2c5d73ded8378a5834d2f3c3733b66c743d6bc461bb Nov 02 19:59:11.006530 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.006500 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz"] Nov 02 19:59:11.006987 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:59:11.006963 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44782bb2_91ba_4c84_bae8_529bb304390a.slice/crio-00c861aa654802e15a56d98379c775ba654fc13fa626748bb9c726d79875c7b2 WatchSource:0}: Error finding container 00c861aa654802e15a56d98379c775ba654fc13fa626748bb9c726d79875c7b2: Status 404 returned error can't find the container with id 00c861aa654802e15a56d98379c775ba654fc13fa626748bb9c726d79875c7b2 Nov 02 19:59:11.167901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.167793 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" event={"ID":"44782bb2-91ba-4c84-bae8-529bb304390a","Type":"ContainerStarted","Data":"dfe22961548e3472a860885c176fdfda2807587598c26831db57c3cf2a6c135e"} Nov 02 19:59:11.167901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.167871 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" event={"ID":"44782bb2-91ba-4c84-bae8-529bb304390a","Type":"ContainerStarted","Data":"00c861aa654802e15a56d98379c775ba654fc13fa626748bb9c726d79875c7b2"} Nov 02 19:59:11.167901 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.167901 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:11.170310 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.170257 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" event={"ID":"c435f75f-ad8b-40fb-b461-fbf070cada9d","Type":"ContainerStarted","Data":"f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf"} Nov 02 19:59:11.170461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.170360 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" event={"ID":"c435f75f-ad8b-40fb-b461-fbf070cada9d","Type":"ContainerStarted","Data":"1fad4d1ca0f8132a9d5fd2c5d73ded8378a5834d2f3c3733b66c743d6bc461bb"} Nov 02 19:59:11.170461 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.170390 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:11.273726 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.273668 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" podStartSLOduration=2.273654257 podStartE2EDuration="2.273654257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:11.271875073 +0000 UTC m=+1857.220530913" watchObservedRunningTime="2026-11-02 19:59:11.273654257 +0000 UTC m=+1857.222310095" Nov 02 19:59:11.383365 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:11.380124 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" podStartSLOduration=1.380106509 podStartE2EDuration="1.380106509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:11.376595588 +0000 UTC m=+1857.325251426" watchObservedRunningTime="2026-11-02 19:59:11.380106509 +0000 UTC m=+1857.328762348" Nov 02 19:59:14.754485 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:59:14.754443 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:59:21.047114 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:21.047071 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t"] Nov 02 19:59:21.092847 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:21.092809 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774023d1-gq84t"] Nov 02 19:59:22.185737 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:22.185702 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96100327-a10a-4fcf-a9e0-e7457db3a986" path="/var/lib/kubelet/pods/96100327-a10a-4fcf-a9e0-e7457db3a986/volumes" Nov 02 19:59:22.186155 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:22.186111 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 19:59:22.186220 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:22.186170 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 19:59:44.755777 ip-10-0-132-174 kubenswrapper[4966]: E1102 19:59:44.755735 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 19:59:56.398645 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.398605 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f"] Nov 02 19:59:56.405362 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.405336 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.464604 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.464560 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f"] Nov 02 19:59:56.496880 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.496842 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ab11604-dockercfg-rztvn\"" Nov 02 19:59:56.498207 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.498182 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ab11604\"" Nov 02 19:59:56.503231 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.503183 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnld\" (UniqueName: \"kubernetes.io/projected/27e043f8-4ed5-423d-912a-d7b3bdeb328f-kube-api-access-4mnld\") pod \"payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.503561 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.503532 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e043f8-4ed5-423d-912a-d7b3bdeb328f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.631240 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.631201 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnld\" (UniqueName: \"kubernetes.io/projected/27e043f8-4ed5-423d-912a-d7b3bdeb328f-kube-api-access-4mnld\") pod \"payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.631775 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.631751 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e043f8-4ed5-423d-912a-d7b3bdeb328f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.633498 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.633251 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e043f8-4ed5-423d-912a-d7b3bdeb328f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.646051 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.646018 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9"] Nov 02 19:59:56.674891 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.660661 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:56.734135 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.733519 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9"] Nov 02 19:59:56.737008 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.736978 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnld\" (UniqueName: \"kubernetes.io/projected/27e043f8-4ed5-423d-912a-d7b3bdeb328f-kube-api-access-4mnld\") pod \"payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.737391 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.737368 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:56.838438 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.838170 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vvc\" (UniqueName: \"kubernetes.io/projected/4ba52fba-eeac-4146-9792-a30af53544ee-kube-api-access-b8vvc\") pod \"payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:56.838876 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.838500 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba52fba-eeac-4146-9792-a30af53544ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:56.940104 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.940019 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba52fba-eeac-4146-9792-a30af53544ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:56.940259 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.940189 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vvc\" (UniqueName: \"kubernetes.io/projected/4ba52fba-eeac-4146-9792-a30af53544ee-kube-api-access-b8vvc\") pod \"payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:56.941191 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.941157 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba52fba-eeac-4146-9792-a30af53544ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:56.992984 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:56.992949 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vvc\" (UniqueName: \"kubernetes.io/projected/4ba52fba-eeac-4146-9792-a30af53544ee-kube-api-access-b8vvc\") pod \"payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:57.118736 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.118701 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:57.154132 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.154096 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f"] Nov 02 19:59:57.158943 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:59:57.158722 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e043f8_4ed5_423d_912a_d7b3bdeb328f.slice/crio-0d55f7ac313f652beff5a70318d512bd163fe3a88f89c2715741aa8a2caac77f WatchSource:0}: Error finding container 0d55f7ac313f652beff5a70318d512bd163fe3a88f89c2715741aa8a2caac77f: Status 404 returned error can't find the container with id 0d55f7ac313f652beff5a70318d512bd163fe3a88f89c2715741aa8a2caac77f Nov 02 19:59:57.508238 ip-10-0-132-174 kubenswrapper[4966]: W1102 19:59:57.508200 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba52fba_eeac_4146_9792_a30af53544ee.slice/crio-f4808dab65170806885916cb63ba6ce2c1f13fcd23d18537787d5cce1fd7f0e2 WatchSource:0}: Error finding container f4808dab65170806885916cb63ba6ce2c1f13fcd23d18537787d5cce1fd7f0e2: Status 404 returned error can't find the container with id f4808dab65170806885916cb63ba6ce2c1f13fcd23d18537787d5cce1fd7f0e2 Nov 02 19:59:57.508644 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.508275 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9"] Nov 02 19:59:57.809996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.809899 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" event={"ID":"4ba52fba-eeac-4146-9792-a30af53544ee","Type":"ContainerStarted","Data":"afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558"} Nov 02 19:59:57.809996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.809957 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" event={"ID":"4ba52fba-eeac-4146-9792-a30af53544ee","Type":"ContainerStarted","Data":"f4808dab65170806885916cb63ba6ce2c1f13fcd23d18537787d5cce1fd7f0e2"} Nov 02 19:59:57.809996 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.809984 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 19:59:57.811706 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.811673 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" event={"ID":"27e043f8-4ed5-423d-912a-d7b3bdeb328f","Type":"ContainerStarted","Data":"ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa"} Nov 02 19:59:57.811860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.811724 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" event={"ID":"27e043f8-4ed5-423d-912a-d7b3bdeb328f","Type":"ContainerStarted","Data":"0d55f7ac313f652beff5a70318d512bd163fe3a88f89c2715741aa8a2caac77f"} Nov 02 19:59:57.811860 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.811751 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 19:59:57.919217 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:57.919162 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" podStartSLOduration=1.91914746 podStartE2EDuration="1.91914746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:57.91475005 +0000 UTC m=+1903.863405928" watchObservedRunningTime="2026-11-02 19:59:57.91914746 +0000 UTC m=+1903.867803298" Nov 02 19:59:58.021921 ip-10-0-132-174 kubenswrapper[4966]: I1102 19:59:58.021861 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" podStartSLOduration=2.021843355 podStartE2EDuration="2.021843355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:58.019160527 +0000 UTC m=+1903.967816365" watchObservedRunningTime="2026-11-02 19:59:58.021843355 +0000 UTC m=+1903.970499195" Nov 02 20:00:00.166787 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.166748 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7"] Nov 02 20:00:00.172633 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.172602 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.188009 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.187979 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt"] Nov 02 20:00:00.192332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.192313 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h"] Nov 02 20:00:00.192428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.192367 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.197236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.197217 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.198162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.198104 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4njd\" (UniqueName: \"kubernetes.io/projected/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-kube-api-access-r4njd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.198162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.198132 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnh9s\" (UniqueName: \"kubernetes.io/projected/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-kube-api-access-lnh9s\") pod \"maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.198493 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.198174 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.198493 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.198348 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.210507 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.210489 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6"] Nov 02 20:00:00.214565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.214548 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.214661 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.214569 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf"] Nov 02 20:00:00.219921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.219896 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.220060 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.220037 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4"] Nov 02 20:00:00.224600 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.224582 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b"] Nov 02 20:00:00.224715 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.224686 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.228856 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.228838 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d"] Nov 02 20:00:00.228955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.228940 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.233989 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.233969 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6"] Nov 02 20:00:00.234105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.234074 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.244535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.244510 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl"] Nov 02 20:00:00.244652 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.244632 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.250165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.249934 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.252446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.252423 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:00:00.252729 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.252712 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:00:00.252842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.252828 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:00:00.257540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.257514 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8"] Nov 02 20:00:00.263247 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.263225 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss"] Nov 02 20:00:00.263561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.263544 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.268588 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.268561 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf"] Nov 02 20:00:00.268689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.268636 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:00.274901 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.274703 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7"] Nov 02 20:00:00.275001 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.274793 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.278341 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.278317 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6"] Nov 02 20:00:00.283401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.282847 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h"] Nov 02 20:00:00.287057 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.287028 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt"] Nov 02 20:00:00.301673 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.301635 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b514b7f-b2f8-4dc2-82cc-285eedff1071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.301866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.301705 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d12a6a-61da-4204-b826-75da5bb34463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.301866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.301801 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plr8\" (UniqueName: \"kubernetes.io/projected/058383f3-e72a-4f34-b231-0acbde4b855d-kube-api-access-4plr8\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.301866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.301847 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/bc80a18b-04c3-44d6-809a-230cbde7d0db-kube-api-access-bsd5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.302049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.301892 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.302049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.301939 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flv9n\" (UniqueName: \"kubernetes.io/projected/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-kube-api-access-flv9n\") pod \"maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:00.302049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.301965 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.302049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302036 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96626ff-f0c1-42d2-a7ba-21d659e5313a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.302484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302073 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.302484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302155 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wk9f\" (UniqueName: \"kubernetes.io/projected/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-kube-api-access-5wk9f\") pod \"maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.302484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302184 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknhx\" (UniqueName: \"kubernetes.io/projected/789d89be-2cd9-481f-a5ed-2590ff0c163c-kube-api-access-cknhx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.302484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302227 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phb5\" (UniqueName: \"kubernetes.io/projected/a96626ff-f0c1-42d2-a7ba-21d659e5313a-kube-api-access-7phb5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.302484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302255 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2lr\" (UniqueName: \"kubernetes.io/projected/33e43a40-2677-4be6-9327-2e3424252de1-kube-api-access-dn2lr\") pod \"maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.302484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302288 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hldz\" (UniqueName: \"kubernetes.io/projected/5b514b7f-b2f8-4dc2-82cc-285eedff1071-kube-api-access-8hldz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.302484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.302376 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789d89be-2cd9-481f-a5ed-2590ff0c163c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.303182 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303159 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4njd\" (UniqueName: \"kubernetes.io/projected/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-kube-api-access-r4njd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.303283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303212 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.303283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303265 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnh9s\" (UniqueName: \"kubernetes.io/projected/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-kube-api-access-lnh9s\") pod \"maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.303517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303494 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.303581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303536 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80a18b-04c3-44d6-809a-230cbde7d0db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.303642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303596 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9c7\" (UniqueName: \"kubernetes.io/projected/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-kube-api-access-lc9c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.303642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303625 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058383f3-e72a-4f34-b231-0acbde4b855d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.303762 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303652 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7bvq\" (UniqueName: \"kubernetes.io/projected/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-kube-api-access-j7bvq\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.303762 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303692 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jtn\" (UniqueName: \"kubernetes.io/projected/c0d12a6a-61da-4204-b826-75da5bb34463-kube-api-access-n2jtn\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.303762 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303721 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.303762 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303744 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:00.303980 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.303794 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e43a40-2677-4be6-9327-2e3424252de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.304713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.304690 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.316205 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.315046 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4"] Nov 02 20:00:00.360710 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.360674 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl"] Nov 02 20:00:00.367521 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.363182 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4njd\" (UniqueName: \"kubernetes.io/projected/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-kube-api-access-r4njd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.367703 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.364087 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnh9s\" (UniqueName: \"kubernetes.io/projected/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-kube-api-access-lnh9s\") pod \"maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.400330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.397800 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d"] Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.412366 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80a18b-04c3-44d6-809a-230cbde7d0db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.412641 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9c7\" (UniqueName: \"kubernetes.io/projected/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-kube-api-access-lc9c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.412689 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058383f3-e72a-4f34-b231-0acbde4b855d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.412900 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bvq\" (UniqueName: \"kubernetes.io/projected/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-kube-api-access-j7bvq\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.412957 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jtn\" (UniqueName: \"kubernetes.io/projected/c0d12a6a-61da-4204-b826-75da5bb34463-kube-api-access-n2jtn\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.412998 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413028 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413086 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e43a40-2677-4be6-9327-2e3424252de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413149 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b514b7f-b2f8-4dc2-82cc-285eedff1071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413488 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80a18b-04c3-44d6-809a-230cbde7d0db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413545 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d12a6a-61da-4204-b826-75da5bb34463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413595 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plr8\" (UniqueName: \"kubernetes.io/projected/058383f3-e72a-4f34-b231-0acbde4b855d-kube-api-access-4plr8\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413637 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/bc80a18b-04c3-44d6-809a-230cbde7d0db-kube-api-access-bsd5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413886 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.413965 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flv9n\" (UniqueName: \"kubernetes.io/projected/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-kube-api-access-flv9n\") pod \"maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.414002 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d12a6a-61da-4204-b826-75da5bb34463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.414003 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.414097 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96626ff-f0c1-42d2-a7ba-21d659e5313a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.414743 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wk9f\" (UniqueName: \"kubernetes.io/projected/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-kube-api-access-5wk9f\") pod \"maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.414923 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96626ff-f0c1-42d2-a7ba-21d659e5313a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.414924 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.415403 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.415957 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058383f3-e72a-4f34-b231-0acbde4b855d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.416433 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.416875 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.417403 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e43a40-2677-4be6-9327-2e3424252de1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.417460 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknhx\" (UniqueName: \"kubernetes.io/projected/789d89be-2cd9-481f-a5ed-2590ff0c163c-kube-api-access-cknhx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.417508 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phb5\" (UniqueName: \"kubernetes.io/projected/a96626ff-f0c1-42d2-a7ba-21d659e5313a-kube-api-access-7phb5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.417720 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2lr\" (UniqueName: \"kubernetes.io/projected/33e43a40-2677-4be6-9327-2e3424252de1-kube-api-access-dn2lr\") pod \"maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.417764 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b514b7f-b2f8-4dc2-82cc-285eedff1071-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.417779 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hldz\" (UniqueName: \"kubernetes.io/projected/5b514b7f-b2f8-4dc2-82cc-285eedff1071-kube-api-access-8hldz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.417834 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789d89be-2cd9-481f-a5ed-2590ff0c163c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.420391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.418738 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789d89be-2cd9-481f-a5ed-2590ff0c163c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.424499 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.424476 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss"] Nov 02 20:00:00.431137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.431114 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf"] Nov 02 20:00:00.452278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.452253 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8"] Nov 02 20:00:00.458985 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.458958 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b"] Nov 02 20:00:00.466669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.466355 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf"] Nov 02 20:00:00.477338 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.477014 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wk9f\" (UniqueName: \"kubernetes.io/projected/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-kube-api-access-5wk9f\") pod \"maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.478536 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.478431 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6"] Nov 02 20:00:00.483093 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.483061 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknhx\" (UniqueName: \"kubernetes.io/projected/789d89be-2cd9-481f-a5ed-2590ff0c163c-kube-api-access-cknhx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.483607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.483581 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bvq\" (UniqueName: \"kubernetes.io/projected/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-kube-api-access-j7bvq\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.484170 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.484146 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hldz\" (UniqueName: \"kubernetes.io/projected/5b514b7f-b2f8-4dc2-82cc-285eedff1071-kube-api-access-8hldz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.486191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.486134 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jtn\" (UniqueName: \"kubernetes.io/projected/c0d12a6a-61da-4204-b826-75da5bb34463-kube-api-access-n2jtn\") pod \"maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.486922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.486895 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phb5\" (UniqueName: \"kubernetes.io/projected/a96626ff-f0c1-42d2-a7ba-21d659e5313a-kube-api-access-7phb5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.487572 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.487524 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/bc80a18b-04c3-44d6-809a-230cbde7d0db-kube-api-access-bsd5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.489110 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.489089 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2lr\" (UniqueName: \"kubernetes.io/projected/33e43a40-2677-4be6-9327-2e3424252de1-kube-api-access-dn2lr\") pod \"maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.489790 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.489760 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9c7\" (UniqueName: \"kubernetes.io/projected/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-kube-api-access-lc9c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.490397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.490381 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:00.492809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.492787 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flv9n\" (UniqueName: \"kubernetes.io/projected/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-kube-api-access-flv9n\") pod \"maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:00.493283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.493266 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plr8\" (UniqueName: \"kubernetes.io/projected/058383f3-e72a-4f34-b231-0acbde4b855d-kube-api-access-4plr8\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.508899 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.508871 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:00.516888 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.516725 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:00.528937 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.528913 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:00.539216 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.539145 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:00.552340 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.552157 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:00.561904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.561534 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:00.582916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.579569 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:00.591004 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.590971 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:00.617560 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.617477 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:00.728911 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.727581 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:00.779334 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.768921 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:00.797413 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:00.779880 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:01.251018 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.250985 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6"] Nov 02 20:00:01.314199 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.314163 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7"] Nov 02 20:00:01.316732 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.316706 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode015108d_dcbb_4fec_a3a9_3a7fd0f8178e.slice/crio-ce028bc164d4f937a69ee2e173a483b7bbcacec59f83ced34a61fb4d243c61de WatchSource:0}: Error finding container ce028bc164d4f937a69ee2e173a483b7bbcacec59f83ced34a61fb4d243c61de: Status 404 returned error can't find the container with id ce028bc164d4f937a69ee2e173a483b7bbcacec59f83ced34a61fb4d243c61de Nov 02 20:00:01.331520 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.331196 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h"] Nov 02 20:00:01.343763 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.343730 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058383f3_e72a_4f34_b231_0acbde4b855d.slice/crio-6ef731e1954371f492e5d699b4b8eaf5de064f8093bc92202207e8ae5dd55cac WatchSource:0}: Error finding container 6ef731e1954371f492e5d699b4b8eaf5de064f8093bc92202207e8ae5dd55cac: Status 404 returned error can't find the container with id 6ef731e1954371f492e5d699b4b8eaf5de064f8093bc92202207e8ae5dd55cac Nov 02 20:00:01.345035 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.345009 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt"] Nov 02 20:00:01.475109 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.475073 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789d89be_2cd9_481f_a5ed_2590ff0c163c.slice/crio-4365057c2d916cb0c9487fcc943070e08c33c8ce97745b96e8b4e329c4b39fb0 WatchSource:0}: Error finding container 4365057c2d916cb0c9487fcc943070e08c33c8ce97745b96e8b4e329c4b39fb0: Status 404 returned error can't find the container with id 4365057c2d916cb0c9487fcc943070e08c33c8ce97745b96e8b4e329c4b39fb0 Nov 02 20:00:01.475473 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.475450 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b"] Nov 02 20:00:01.518559 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.518534 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl"] Nov 02 20:00:01.519977 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.519950 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a43c6d_a71c_4a48_a5b8_a16c10476d86.slice/crio-6dd0a09f9ec88bfb8a213c8980164b2ac1c5828165461c457d8cce0c2215c7b5 WatchSource:0}: Error finding container 6dd0a09f9ec88bfb8a213c8980164b2ac1c5828165461c457d8cce0c2215c7b5: Status 404 returned error can't find the container with id 6dd0a09f9ec88bfb8a213c8980164b2ac1c5828165461c457d8cce0c2215c7b5 Nov 02 20:00:01.537632 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.537554 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4"] Nov 02 20:00:01.538965 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.538931 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d12a6a_61da_4204_b826_75da5bb34463.slice/crio-5178eca9855b2edafdc97783c996d9db58c20e8c8f35d7cd5b0e633e0d7175ec WatchSource:0}: Error finding container 5178eca9855b2edafdc97783c996d9db58c20e8c8f35d7cd5b0e633e0d7175ec: Status 404 returned error can't find the container with id 5178eca9855b2edafdc97783c996d9db58c20e8c8f35d7cd5b0e633e0d7175ec Nov 02 20:00:01.546626 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.546550 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d"] Nov 02 20:00:01.551651 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.551081 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78d71ed_3361_4b0e_98fc_c3cb9f174a02.slice/crio-c52a07bd0c51a04cd2be114298d3cb34759988977a43b5b270dc9dc9f34231e6 WatchSource:0}: Error finding container c52a07bd0c51a04cd2be114298d3cb34759988977a43b5b270dc9dc9f34231e6: Status 404 returned error can't find the container with id c52a07bd0c51a04cd2be114298d3cb34759988977a43b5b270dc9dc9f34231e6 Nov 02 20:00:01.553898 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.553870 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc80a18b_04c3_44d6_809a_230cbde7d0db.slice/crio-f4ed032dfae07c552d50fea1063c1e7eaa5643a544275def815393581d92d4f8 WatchSource:0}: Error finding container f4ed032dfae07c552d50fea1063c1e7eaa5643a544275def815393581d92d4f8: Status 404 returned error can't find the container with id f4ed032dfae07c552d50fea1063c1e7eaa5643a544275def815393581d92d4f8 Nov 02 20:00:01.566663 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.566636 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf"] Nov 02 20:00:01.588425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.588394 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8"] Nov 02 20:00:01.594413 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.594389 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf"] Nov 02 20:00:01.601062 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.601034 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e43a40_2677_4be6_9327_2e3424252de1.slice/crio-b8ca07c649aaa3b69c4b5e6f417b943454186d2e09245a1e4ffa27818c2b00fc WatchSource:0}: Error finding container b8ca07c649aaa3b69c4b5e6f417b943454186d2e09245a1e4ffa27818c2b00fc: Status 404 returned error can't find the container with id b8ca07c649aaa3b69c4b5e6f417b943454186d2e09245a1e4ffa27818c2b00fc Nov 02 20:00:01.601643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.601620 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss"] Nov 02 20:00:01.603630 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:01.603606 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149c68c2_f8a0_4d24_8e66_e2abcfc3aacf.slice/crio-b9cef6cf3cdb34c3148f74f0436dbe47f7a3bf1d2bf30744341dfe65006322d6 WatchSource:0}: Error finding container b9cef6cf3cdb34c3148f74f0436dbe47f7a3bf1d2bf30744341dfe65006322d6: Status 404 returned error can't find the container with id b9cef6cf3cdb34c3148f74f0436dbe47f7a3bf1d2bf30744341dfe65006322d6 Nov 02 20:00:01.654562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.654527 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6"] Nov 02 20:00:01.884180 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.883466 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" event={"ID":"a78d71ed-3361-4b0e-98fc-c3cb9f174a02","Type":"ContainerStarted","Data":"c52a07bd0c51a04cd2be114298d3cb34759988977a43b5b270dc9dc9f34231e6"} Nov 02 20:00:01.885861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.885831 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerStarted","Data":"6dd0a09f9ec88bfb8a213c8980164b2ac1c5828165461c457d8cce0c2215c7b5"} Nov 02 20:00:01.888017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.887980 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" event={"ID":"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4","Type":"ContainerStarted","Data":"95ac553b5499a2069cb4dd12b6c4023bacd0874ef1b45ee2a02f3da7d046120d"} Nov 02 20:00:01.889530 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.889265 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" event={"ID":"789d89be-2cd9-481f-a5ed-2590ff0c163c","Type":"ContainerStarted","Data":"4365057c2d916cb0c9487fcc943070e08c33c8ce97745b96e8b4e329c4b39fb0"} Nov 02 20:00:01.891442 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.891413 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerStarted","Data":"75f582febe9b172e4d1303d2c6e51b4df96c4b533f84cfb6d8ada8eee41ca6ec"} Nov 02 20:00:01.891532 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.891454 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerStarted","Data":"e9073d98bbcb4d4203f6366ceacf05f54a646fd5dadd276ca750f9e61f0a101b"} Nov 02 20:00:01.893468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.893446 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerStarted","Data":"da2bce5c7dbf75a88bc6e0ced30e3f6e5f556332dd05d387aa47066244b6472b"} Nov 02 20:00:01.894973 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.894949 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" event={"ID":"058383f3-e72a-4f34-b231-0acbde4b855d","Type":"ContainerStarted","Data":"6ef731e1954371f492e5d699b4b8eaf5de064f8093bc92202207e8ae5dd55cac"} Nov 02 20:00:01.896376 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.896328 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" event={"ID":"5b514b7f-b2f8-4dc2-82cc-285eedff1071","Type":"ContainerStarted","Data":"ba3e84fe47934756080b786604392ba9f538933d65aaacb15dc0daee8c1cc7f6"} Nov 02 20:00:01.898160 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.898133 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" event={"ID":"33e43a40-2677-4be6-9327-2e3424252de1","Type":"ContainerStarted","Data":"b8ca07c649aaa3b69c4b5e6f417b943454186d2e09245a1e4ffa27818c2b00fc"} Nov 02 20:00:01.899795 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.899774 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerStarted","Data":"f4ed032dfae07c552d50fea1063c1e7eaa5643a544275def815393581d92d4f8"} Nov 02 20:00:01.901642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.901611 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" event={"ID":"c0d12a6a-61da-4204-b826-75da5bb34463","Type":"ContainerStarted","Data":"5178eca9855b2edafdc97783c996d9db58c20e8c8f35d7cd5b0e633e0d7175ec"} Nov 02 20:00:01.903379 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.903203 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerStarted","Data":"b9cef6cf3cdb34c3148f74f0436dbe47f7a3bf1d2bf30744341dfe65006322d6"} Nov 02 20:00:01.904755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:01.904736 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" event={"ID":"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e","Type":"ContainerStarted","Data":"ce028bc164d4f937a69ee2e173a483b7bbcacec59f83ced34a61fb4d243c61de"} Nov 02 20:00:02.007334 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.007169 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" podStartSLOduration=2.00715061 podStartE2EDuration="2.00715061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:02.003416741 +0000 UTC m=+1907.952072577" watchObservedRunningTime="2026-11-02 20:00:02.00715061 +0000 UTC m=+1907.955806450" Nov 02 20:00:02.915782 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.915749 4966 generic.go:356] "Generic (PLEG): container finished" podID="c0d12a6a-61da-4204-b826-75da5bb34463" containerID="db0ed5e00047e155f5e5c26edf062f3691b67926e0514ac6245e8fd14385dc6f" exitCode=60 Nov 02 20:00:02.916158 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.915834 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" event={"ID":"c0d12a6a-61da-4204-b826-75da5bb34463","Type":"ContainerDied","Data":"db0ed5e00047e155f5e5c26edf062f3691b67926e0514ac6245e8fd14385dc6f"} Nov 02 20:00:02.916158 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.916121 4966 scope.go:122] "RemoveContainer" containerID="db0ed5e00047e155f5e5c26edf062f3691b67926e0514ac6245e8fd14385dc6f" Nov 02 20:00:02.917760 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.917736 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerStarted","Data":"1466d3ea0f66dbfe798e9b813363eabd713a46e6f48a38983a9d29bbd8c2b3e3"} Nov 02 20:00:02.919630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.919604 4966 generic.go:356] "Generic (PLEG): container finished" podID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerID="b8f7904818820dbf4ba48a4fbfa6014ec56e83491d107649ad712db1062c3c83" exitCode=60 Nov 02 20:00:02.919737 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.919706 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" event={"ID":"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e","Type":"ContainerDied","Data":"b8f7904818820dbf4ba48a4fbfa6014ec56e83491d107649ad712db1062c3c83"} Nov 02 20:00:02.920063 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.919926 4966 scope.go:122] "RemoveContainer" containerID="b8f7904818820dbf4ba48a4fbfa6014ec56e83491d107649ad712db1062c3c83" Nov 02 20:00:02.921456 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.921436 4966 generic.go:356] "Generic (PLEG): container finished" podID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerID="846a9e9f414858210d34b75c52c5592d2a7bf413ae6707fb816b4ad20d127f73" exitCode=60 Nov 02 20:00:02.921537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.921505 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" event={"ID":"a78d71ed-3361-4b0e-98fc-c3cb9f174a02","Type":"ContainerDied","Data":"846a9e9f414858210d34b75c52c5592d2a7bf413ae6707fb816b4ad20d127f73"} Nov 02 20:00:02.921755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.921741 4966 scope.go:122] "RemoveContainer" containerID="846a9e9f414858210d34b75c52c5592d2a7bf413ae6707fb816b4ad20d127f73" Nov 02 20:00:02.923155 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.923131 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerStarted","Data":"fee1949195f77a828a47eb3ab427dd45ff698483e3b53cea07cafe657e754a18"} Nov 02 20:00:02.924820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.924799 4966 generic.go:356] "Generic (PLEG): container finished" podID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerID="4a2da3158470a51e1557bfab2ba753227aa25782b641ffb0a15f1ed199aab9ba" exitCode=60 Nov 02 20:00:02.924918 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.924858 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" event={"ID":"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4","Type":"ContainerDied","Data":"4a2da3158470a51e1557bfab2ba753227aa25782b641ffb0a15f1ed199aab9ba"} Nov 02 20:00:02.925055 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.925041 4966 scope.go:122] "RemoveContainer" containerID="4a2da3158470a51e1557bfab2ba753227aa25782b641ffb0a15f1ed199aab9ba" Nov 02 20:00:02.926542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.926524 4966 generic.go:356] "Generic (PLEG): container finished" podID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerID="d943b41318e21973a36ac6c4ad5f0f92f30264a1b83d346bbb32e9be09edea4b" exitCode=60 Nov 02 20:00:02.926642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.926600 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" event={"ID":"789d89be-2cd9-481f-a5ed-2590ff0c163c","Type":"ContainerDied","Data":"d943b41318e21973a36ac6c4ad5f0f92f30264a1b83d346bbb32e9be09edea4b"} Nov 02 20:00:02.926775 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.926761 4966 scope.go:122] "RemoveContainer" containerID="d943b41318e21973a36ac6c4ad5f0f92f30264a1b83d346bbb32e9be09edea4b" Nov 02 20:00:02.928237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.928219 4966 generic.go:356] "Generic (PLEG): container finished" podID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerID="75f582febe9b172e4d1303d2c6e51b4df96c4b533f84cfb6d8ada8eee41ca6ec" exitCode=60 Nov 02 20:00:02.928367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.928294 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerDied","Data":"75f582febe9b172e4d1303d2c6e51b4df96c4b533f84cfb6d8ada8eee41ca6ec"} Nov 02 20:00:02.928564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.928548 4966 scope.go:122] "RemoveContainer" containerID="75f582febe9b172e4d1303d2c6e51b4df96c4b533f84cfb6d8ada8eee41ca6ec" Nov 02 20:00:02.930983 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.930906 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerStarted","Data":"70d1b01980afd22c37a80f3eb1d55c2bbda79455e771a707849675ab0d7f19cb"} Nov 02 20:00:02.932739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.932719 4966 generic.go:356] "Generic (PLEG): container finished" podID="058383f3-e72a-4f34-b231-0acbde4b855d" containerID="b97ec83f1f0ec0dff43761b18ece1318e6038ce9e3b8599f1d8252172b359696" exitCode=60 Nov 02 20:00:02.932822 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.932781 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" event={"ID":"058383f3-e72a-4f34-b231-0acbde4b855d","Type":"ContainerDied","Data":"b97ec83f1f0ec0dff43761b18ece1318e6038ce9e3b8599f1d8252172b359696"} Nov 02 20:00:02.933048 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.933026 4966 scope.go:122] "RemoveContainer" containerID="b97ec83f1f0ec0dff43761b18ece1318e6038ce9e3b8599f1d8252172b359696" Nov 02 20:00:02.934842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.934822 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" event={"ID":"5b514b7f-b2f8-4dc2-82cc-285eedff1071","Type":"ContainerStarted","Data":"17437a69cffb4a9247c3b82a0a0aa34f89fdfa67eccbf99ec90c7460b517387e"} Nov 02 20:00:02.940757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.938383 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" event={"ID":"33e43a40-2677-4be6-9327-2e3424252de1","Type":"ContainerStarted","Data":"5b9e08528bd5df47bec0db82cf25d43ffa4409eddafb85fe4e5bd90080698926"} Nov 02 20:00:02.940757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:02.940263 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerStarted","Data":"761fda188f40d6e5bc6183a9d3e24a82d15f5e70e0bcc340892822d0208159b0"} Nov 02 20:00:03.341019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.340955 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" podStartSLOduration=3.340935749 podStartE2EDuration="3.340935749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:03.334028525 +0000 UTC m=+1909.282684388" watchObservedRunningTime="2026-11-02 20:00:03.340935749 +0000 UTC m=+1909.289591599" Nov 02 20:00:03.439202 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.439140 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" podStartSLOduration=3.439120924 podStartE2EDuration="3.439120924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:03.438170151 +0000 UTC m=+1909.386826026" watchObservedRunningTime="2026-11-02 20:00:03.439120924 +0000 UTC m=+1909.387776764" Nov 02 20:00:03.548563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.548502 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" podStartSLOduration=3.548482718 podStartE2EDuration="3.548482718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:03.543727617 +0000 UTC m=+1909.492383457" watchObservedRunningTime="2026-11-02 20:00:03.548482718 +0000 UTC m=+1909.497138558" Nov 02 20:00:03.955187 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.955059 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" event={"ID":"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e","Type":"ContainerStarted","Data":"8212002fe57845ae968554db4bf2c8cb55fbd6fc24fb389c53cb3d34167c8f8b"} Nov 02 20:00:03.957652 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.957623 4966 generic.go:356] "Generic (PLEG): container finished" podID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerID="fee1949195f77a828a47eb3ab427dd45ff698483e3b53cea07cafe657e754a18" exitCode=60 Nov 02 20:00:03.957818 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.957704 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerDied","Data":"fee1949195f77a828a47eb3ab427dd45ff698483e3b53cea07cafe657e754a18"} Nov 02 20:00:03.958045 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.958027 4966 scope.go:122] "RemoveContainer" containerID="fee1949195f77a828a47eb3ab427dd45ff698483e3b53cea07cafe657e754a18" Nov 02 20:00:03.960285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.960263 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" event={"ID":"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4","Type":"ContainerStarted","Data":"a31777a88c64f43f4d03dcb939e2c30df0ff8c546b627bba9b3aea3aec33684b"} Nov 02 20:00:03.962970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.962948 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" event={"ID":"789d89be-2cd9-481f-a5ed-2590ff0c163c","Type":"ContainerStarted","Data":"727b2d3ccd8e8944a8217fac7b0a2d7f2ce2e104cac4a0687b22ee00728e90d7"} Nov 02 20:00:03.965163 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.965133 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerStarted","Data":"0471fde7494e8414af417fa95f89ca95bd4b91484bf4f0d5820e1925dd1a4e88"} Nov 02 20:00:03.967390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.967255 4966 generic.go:356] "Generic (PLEG): container finished" podID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerID="70d1b01980afd22c37a80f3eb1d55c2bbda79455e771a707849675ab0d7f19cb" exitCode=60 Nov 02 20:00:03.967390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.967286 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerDied","Data":"70d1b01980afd22c37a80f3eb1d55c2bbda79455e771a707849675ab0d7f19cb"} Nov 02 20:00:03.967970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.967952 4966 scope.go:122] "RemoveContainer" containerID="70d1b01980afd22c37a80f3eb1d55c2bbda79455e771a707849675ab0d7f19cb" Nov 02 20:00:03.969666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.969646 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" event={"ID":"058383f3-e72a-4f34-b231-0acbde4b855d","Type":"ContainerStarted","Data":"dcc832b3b2cc5a68bfa4136aec8fda4ace641a92cf614a02825934aff8bf0f6e"} Nov 02 20:00:03.972269 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.972237 4966 generic.go:356] "Generic (PLEG): container finished" podID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerID="17437a69cffb4a9247c3b82a0a0aa34f89fdfa67eccbf99ec90c7460b517387e" exitCode=60 Nov 02 20:00:03.972527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.972485 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" podStartSLOduration=3.9724711299999997 podStartE2EDuration="3.97247113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:03.97022387 +0000 UTC m=+1909.918879710" watchObservedRunningTime="2026-11-02 20:00:03.97247113 +0000 UTC m=+1909.921126971" Nov 02 20:00:03.972699 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.972680 4966 scope.go:122] "RemoveContainer" containerID="17437a69cffb4a9247c3b82a0a0aa34f89fdfa67eccbf99ec90c7460b517387e" Nov 02 20:00:03.973142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.972547 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" event={"ID":"5b514b7f-b2f8-4dc2-82cc-285eedff1071","Type":"ContainerDied","Data":"17437a69cffb4a9247c3b82a0a0aa34f89fdfa67eccbf99ec90c7460b517387e"} Nov 02 20:00:03.975204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.975177 4966 generic.go:356] "Generic (PLEG): container finished" podID="33e43a40-2677-4be6-9327-2e3424252de1" containerID="5b9e08528bd5df47bec0db82cf25d43ffa4409eddafb85fe4e5bd90080698926" exitCode=60 Nov 02 20:00:03.975350 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.975314 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" event={"ID":"33e43a40-2677-4be6-9327-2e3424252de1","Type":"ContainerDied","Data":"5b9e08528bd5df47bec0db82cf25d43ffa4409eddafb85fe4e5bd90080698926"} Nov 02 20:00:03.975626 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.975609 4966 scope.go:122] "RemoveContainer" containerID="5b9e08528bd5df47bec0db82cf25d43ffa4409eddafb85fe4e5bd90080698926" Nov 02 20:00:03.978799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.978776 4966 generic.go:356] "Generic (PLEG): container finished" podID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerID="761fda188f40d6e5bc6183a9d3e24a82d15f5e70e0bcc340892822d0208159b0" exitCode=60 Nov 02 20:00:03.978904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.978873 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerDied","Data":"761fda188f40d6e5bc6183a9d3e24a82d15f5e70e0bcc340892822d0208159b0"} Nov 02 20:00:03.979204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.979187 4966 scope.go:122] "RemoveContainer" containerID="761fda188f40d6e5bc6183a9d3e24a82d15f5e70e0bcc340892822d0208159b0" Nov 02 20:00:03.981860 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.981835 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" event={"ID":"c0d12a6a-61da-4204-b826-75da5bb34463","Type":"ContainerStarted","Data":"677821b2fe76c633384222ba611bb1b9535a0bb363124988b064abbc227f9c47"} Nov 02 20:00:03.983941 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.983920 4966 generic.go:356] "Generic (PLEG): container finished" podID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerID="1466d3ea0f66dbfe798e9b813363eabd713a46e6f48a38983a9d29bbd8c2b3e3" exitCode=60 Nov 02 20:00:03.984033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.983954 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerDied","Data":"1466d3ea0f66dbfe798e9b813363eabd713a46e6f48a38983a9d29bbd8c2b3e3"} Nov 02 20:00:03.984156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:03.984141 4966 scope.go:122] "RemoveContainer" containerID="1466d3ea0f66dbfe798e9b813363eabd713a46e6f48a38983a9d29bbd8c2b3e3" Nov 02 20:00:04.079115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.078595 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" podStartSLOduration=4.078576732 podStartE2EDuration="4.078576732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:04.075144143 +0000 UTC m=+1910.023799982" watchObservedRunningTime="2026-11-02 20:00:04.078576732 +0000 UTC m=+1910.027232573" Nov 02 20:00:04.291384 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.291334 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" podStartSLOduration=4.291319523 podStartE2EDuration="4.291319523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:04.285432807 +0000 UTC m=+1910.234088652" watchObservedRunningTime="2026-11-02 20:00:04.291319523 +0000 UTC m=+1910.239975376" Nov 02 20:00:04.393210 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.393154 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" podStartSLOduration=4.393137463 podStartE2EDuration="4.393137463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:04.390660436 +0000 UTC m=+1910.339316269" watchObservedRunningTime="2026-11-02 20:00:04.393137463 +0000 UTC m=+1910.341793346" Nov 02 20:00:04.602317 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.602245 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" podStartSLOduration=4.602229068 podStartE2EDuration="4.602229068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:04.60019322 +0000 UTC m=+1910.548849056" watchObservedRunningTime="2026-11-02 20:00:04.602229068 +0000 UTC m=+1910.550884907" Nov 02 20:00:04.706697 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.706639 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" podStartSLOduration=4.706620168 podStartE2EDuration="4.706620168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:04.704713058 +0000 UTC m=+1910.653368901" watchObservedRunningTime="2026-11-02 20:00:04.706620168 +0000 UTC m=+1910.655276005" Nov 02 20:00:04.831755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.831720 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n"] Nov 02 20:00:04.831943 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.831921 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" containerID="cri-o://07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36" gracePeriod=30 Nov 02 20:00:04.995425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.995393 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerStarted","Data":"3e22992da5e5211322e1fae1717bedb7ed0dfb2d73b863b38414b43bdc30366c"} Nov 02 20:00:04.997555 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.997525 4966 generic.go:356] "Generic (PLEG): container finished" podID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerID="8212002fe57845ae968554db4bf2c8cb55fbd6fc24fb389c53cb3d34167c8f8b" exitCode=60 Nov 02 20:00:04.997686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.997604 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" event={"ID":"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e","Type":"ContainerDied","Data":"8212002fe57845ae968554db4bf2c8cb55fbd6fc24fb389c53cb3d34167c8f8b"} Nov 02 20:00:04.997686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.997662 4966 scope.go:122] "RemoveContainer" containerID="b8f7904818820dbf4ba48a4fbfa6014ec56e83491d107649ad712db1062c3c83" Nov 02 20:00:04.997907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.997888 4966 scope.go:122] "RemoveContainer" containerID="8212002fe57845ae968554db4bf2c8cb55fbd6fc24fb389c53cb3d34167c8f8b" Nov 02 20:00:04.998331 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:04.998270 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7_odh-ai-gateway-infra(e015108d-dcbb-4fec-a3a9-3a7fd0f8178e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" Nov 02 20:00:04.999968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:04.999946 4966 generic.go:356] "Generic (PLEG): container finished" podID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerID="866c1bb595e86b29d80ad034011231aaca594a570f8057c6f5b360117f9ce497" exitCode=60 Nov 02 20:00:05.000077 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.000015 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" event={"ID":"a78d71ed-3361-4b0e-98fc-c3cb9f174a02","Type":"ContainerDied","Data":"866c1bb595e86b29d80ad034011231aaca594a570f8057c6f5b360117f9ce497"} Nov 02 20:00:05.000397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.000378 4966 scope.go:122] "RemoveContainer" containerID="866c1bb595e86b29d80ad034011231aaca594a570f8057c6f5b360117f9ce497" Nov 02 20:00:05.000733 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:05.000713 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d_odh-ai-gateway-infra(a78d71ed-3361-4b0e-98fc-c3cb9f174a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" Nov 02 20:00:05.002315 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.002278 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerStarted","Data":"4917f26cda6a06048342c3a0669b244b9df8277f0e9480f524e5506932ede966"} Nov 02 20:00:05.004644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.004617 4966 generic.go:356] "Generic (PLEG): container finished" podID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerID="a31777a88c64f43f4d03dcb939e2c30df0ff8c546b627bba9b3aea3aec33684b" exitCode=60 Nov 02 20:00:05.004748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.004722 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" event={"ID":"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4","Type":"ContainerDied","Data":"a31777a88c64f43f4d03dcb939e2c30df0ff8c546b627bba9b3aea3aec33684b"} Nov 02 20:00:05.004933 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.004917 4966 scope.go:122] "RemoveContainer" containerID="a31777a88c64f43f4d03dcb939e2c30df0ff8c546b627bba9b3aea3aec33684b" Nov 02 20:00:05.005198 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:05.005171 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt_odh-ai-gateway-infra(03e09a4f-2cc9-4044-8ddd-18eedfd04ff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" Nov 02 20:00:05.007026 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.007005 4966 generic.go:356] "Generic (PLEG): container finished" podID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerID="727b2d3ccd8e8944a8217fac7b0a2d7f2ce2e104cac4a0687b22ee00728e90d7" exitCode=60 Nov 02 20:00:05.007144 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.007074 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" event={"ID":"789d89be-2cd9-481f-a5ed-2590ff0c163c","Type":"ContainerDied","Data":"727b2d3ccd8e8944a8217fac7b0a2d7f2ce2e104cac4a0687b22ee00728e90d7"} Nov 02 20:00:05.007377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.007361 4966 scope.go:122] "RemoveContainer" containerID="727b2d3ccd8e8944a8217fac7b0a2d7f2ce2e104cac4a0687b22ee00728e90d7" Nov 02 20:00:05.007620 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:05.007600 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b_odh-ai-gateway-infra(789d89be-2cd9-481f-a5ed-2590ff0c163c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" Nov 02 20:00:05.009860 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.009836 4966 generic.go:356] "Generic (PLEG): container finished" podID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerID="0471fde7494e8414af417fa95f89ca95bd4b91484bf4f0d5820e1925dd1a4e88" exitCode=60 Nov 02 20:00:05.009955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.009937 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerDied","Data":"0471fde7494e8414af417fa95f89ca95bd4b91484bf4f0d5820e1925dd1a4e88"} Nov 02 20:00:05.010248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.010232 4966 scope.go:122] "RemoveContainer" containerID="0471fde7494e8414af417fa95f89ca95bd4b91484bf4f0d5820e1925dd1a4e88" Nov 02 20:00:05.010525 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:05.010503 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6_odh-ai-gateway-infra(42e5bb7d-7b64-4088-9ffd-7ce1ba735894)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" Nov 02 20:00:05.012653 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.012633 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerStarted","Data":"5d012df886c8cacc3bef147272930be8845303c1f43d298a42cc8861251c0572"} Nov 02 20:00:05.014932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.014904 4966 scope.go:122] "RemoveContainer" containerID="846a9e9f414858210d34b75c52c5592d2a7bf413ae6707fb816b4ad20d127f73" Nov 02 20:00:05.015008 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.014950 4966 generic.go:356] "Generic (PLEG): container finished" podID="058383f3-e72a-4f34-b231-0acbde4b855d" containerID="dcc832b3b2cc5a68bfa4136aec8fda4ace641a92cf614a02825934aff8bf0f6e" exitCode=60 Nov 02 20:00:05.015125 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.015108 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" event={"ID":"058383f3-e72a-4f34-b231-0acbde4b855d","Type":"ContainerDied","Data":"dcc832b3b2cc5a68bfa4136aec8fda4ace641a92cf614a02825934aff8bf0f6e"} Nov 02 20:00:05.015254 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.015238 4966 scope.go:122] "RemoveContainer" containerID="dcc832b3b2cc5a68bfa4136aec8fda4ace641a92cf614a02825934aff8bf0f6e" Nov 02 20:00:05.015502 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:05.015480 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h_odh-ai-gateway-infra(058383f3-e72a-4f34-b231-0acbde4b855d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" Nov 02 20:00:05.018279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.018257 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" event={"ID":"5b514b7f-b2f8-4dc2-82cc-285eedff1071","Type":"ContainerStarted","Data":"ca800089ab4091d5d743b45ebfcada7bc07e7b1540d39ed784c42dc775d241d6"} Nov 02 20:00:05.020526 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.020505 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" event={"ID":"33e43a40-2677-4be6-9327-2e3424252de1","Type":"ContainerStarted","Data":"d041e0f84d4378fb9936107f4ba814039f5ac68ac538202a640656ed704c1cb0"} Nov 02 20:00:05.022748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.022728 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerStarted","Data":"85b4d96d4b406f12bc467ef9484c8766de80d73dcfe0954c81425e995f14554f"} Nov 02 20:00:05.029147 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.029121 4966 generic.go:356] "Generic (PLEG): container finished" podID="c0d12a6a-61da-4204-b826-75da5bb34463" containerID="677821b2fe76c633384222ba611bb1b9535a0bb363124988b064abbc227f9c47" exitCode=60 Nov 02 20:00:05.029251 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.029168 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" event={"ID":"c0d12a6a-61da-4204-b826-75da5bb34463","Type":"ContainerDied","Data":"677821b2fe76c633384222ba611bb1b9535a0bb363124988b064abbc227f9c47"} Nov 02 20:00:05.029447 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.029433 4966 scope.go:122] "RemoveContainer" containerID="677821b2fe76c633384222ba611bb1b9535a0bb363124988b064abbc227f9c47" Nov 02 20:00:05.029699 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:05.029673 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4_odh-ai-gateway-infra(c0d12a6a-61da-4204-b826-75da5bb34463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" Nov 02 20:00:05.032616 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.032594 4966 scope.go:122] "RemoveContainer" containerID="4a2da3158470a51e1557bfab2ba753227aa25782b641ffb0a15f1ed199aab9ba" Nov 02 20:00:05.053198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.053167 4966 scope.go:122] "RemoveContainer" containerID="d943b41318e21973a36ac6c4ad5f0f92f30264a1b83d346bbb32e9be09edea4b" Nov 02 20:00:05.072067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.072035 4966 scope.go:122] "RemoveContainer" containerID="75f582febe9b172e4d1303d2c6e51b4df96c4b533f84cfb6d8ada8eee41ca6ec" Nov 02 20:00:05.091161 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.091131 4966 scope.go:122] "RemoveContainer" containerID="b97ec83f1f0ec0dff43761b18ece1318e6038ce9e3b8599f1d8252172b359696" Nov 02 20:00:05.213492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.209908 4966 scope.go:122] "RemoveContainer" containerID="db0ed5e00047e155f5e5c26edf062f3691b67926e0514ac6245e8fd14385dc6f" Nov 02 20:00:05.297378 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.270582 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" podStartSLOduration=5.27056053 podStartE2EDuration="5.27056053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:05.259237549 +0000 UTC m=+1911.207893389" watchObservedRunningTime="2026-11-02 20:00:05.27056053 +0000 UTC m=+1911.219216371" Nov 02 20:00:05.297378 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.278413 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82"] Nov 02 20:00:05.318400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.312440 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.468423 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.466564 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m886n\" (UniqueName: \"kubernetes.io/projected/9f5c6025-a0af-455b-80b6-b38225915f4c-kube-api-access-m886n\") pod \"payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.468423 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.466852 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5c6025-a0af-455b-80b6-b38225915f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.623531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.555503 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9f8f23\"" Nov 02 20:00:05.623531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.556390 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9f8f23-dockercfg-rztmv\"" Nov 02 20:00:05.623531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.564594 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82"] Nov 02 20:00:05.623531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.570053 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m886n\" (UniqueName: \"kubernetes.io/projected/9f5c6025-a0af-455b-80b6-b38225915f4c-kube-api-access-m886n\") pod \"payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.623531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.570242 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5c6025-a0af-455b-80b6-b38225915f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.623531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.572017 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5c6025-a0af-455b-80b6-b38225915f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.643423 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.643338 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl"] Nov 02 20:00:05.644382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.643625 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" podStartSLOduration=5.6436092129999995 podStartE2EDuration="5.643609213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:05.555792163 +0000 UTC m=+1911.504448003" watchObservedRunningTime="2026-11-02 20:00:05.643609213 +0000 UTC m=+1911.592265052" Nov 02 20:00:05.657533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.657013 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:05.661805 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.661630 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl"] Nov 02 20:00:05.677702 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.677628 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m886n\" (UniqueName: \"kubernetes.io/projected/9f5c6025-a0af-455b-80b6-b38225915f4c-kube-api-access-m886n\") pod \"payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.773149 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.773110 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfr9q\" (UniqueName: \"kubernetes.io/projected/029571de-57d3-4dca-91f2-22fd2df6a513-kube-api-access-pfr9q\") pod \"payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:05.773391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.773276 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029571de-57d3-4dca-91f2-22fd2df6a513-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:05.874926 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.874888 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029571de-57d3-4dca-91f2-22fd2df6a513-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:05.875112 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.875098 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfr9q\" (UniqueName: \"kubernetes.io/projected/029571de-57d3-4dca-91f2-22fd2df6a513-kube-api-access-pfr9q\") pod \"payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:05.875863 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.875840 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029571de-57d3-4dca-91f2-22fd2df6a513-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:05.927848 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.927757 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfr9q\" (UniqueName: \"kubernetes.io/projected/029571de-57d3-4dca-91f2-22fd2df6a513-kube-api-access-pfr9q\") pod \"payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:05.953462 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.953157 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:05.980544 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:05.980507 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:06.112439 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.106073 4966 scope.go:122] "RemoveContainer" containerID="8212002fe57845ae968554db4bf2c8cb55fbd6fc24fb389c53cb3d34167c8f8b" Nov 02 20:00:06.112439 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.106432 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7_odh-ai-gateway-infra(e015108d-dcbb-4fec-a3a9-3a7fd0f8178e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" Nov 02 20:00:06.112439 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.111984 4966 scope.go:122] "RemoveContainer" containerID="866c1bb595e86b29d80ad034011231aaca594a570f8057c6f5b360117f9ce497" Nov 02 20:00:06.113224 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.112513 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d_odh-ai-gateway-infra(a78d71ed-3361-4b0e-98fc-c3cb9f174a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" Nov 02 20:00:06.119017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.115528 4966 generic.go:356] "Generic (PLEG): container finished" podID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerID="4917f26cda6a06048342c3a0669b244b9df8277f0e9480f524e5506932ede966" exitCode=60 Nov 02 20:00:06.119017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.115932 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerDied","Data":"4917f26cda6a06048342c3a0669b244b9df8277f0e9480f524e5506932ede966"} Nov 02 20:00:06.119017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.115942 4966 scope.go:122] "RemoveContainer" containerID="4917f26cda6a06048342c3a0669b244b9df8277f0e9480f524e5506932ede966" Nov 02 20:00:06.119017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.115996 4966 scope.go:122] "RemoveContainer" containerID="fee1949195f77a828a47eb3ab427dd45ff698483e3b53cea07cafe657e754a18" Nov 02 20:00:06.119017 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.116452 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl_odh-ai-gateway-infra(a8a43c6d-a71c-4a48-a5b8-a16c10476d86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" Nov 02 20:00:06.119541 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.119235 4966 scope.go:122] "RemoveContainer" containerID="a31777a88c64f43f4d03dcb939e2c30df0ff8c546b627bba9b3aea3aec33684b" Nov 02 20:00:06.119597 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.119559 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt_odh-ai-gateway-infra(03e09a4f-2cc9-4044-8ddd-18eedfd04ff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" Nov 02 20:00:06.122239 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.122080 4966 scope.go:122] "RemoveContainer" containerID="727b2d3ccd8e8944a8217fac7b0a2d7f2ce2e104cac4a0687b22ee00728e90d7" Nov 02 20:00:06.122470 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.122410 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b_odh-ai-gateway-infra(789d89be-2cd9-481f-a5ed-2590ff0c163c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" Nov 02 20:00:06.124605 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.124524 4966 scope.go:122] "RemoveContainer" containerID="0471fde7494e8414af417fa95f89ca95bd4b91484bf4f0d5820e1925dd1a4e88" Nov 02 20:00:06.124979 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.124869 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6_odh-ai-gateway-infra(42e5bb7d-7b64-4088-9ffd-7ce1ba735894)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" Nov 02 20:00:06.133496 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.130536 4966 generic.go:356] "Generic (PLEG): container finished" podID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerID="5d012df886c8cacc3bef147272930be8845303c1f43d298a42cc8861251c0572" exitCode=60 Nov 02 20:00:06.133496 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.130974 4966 scope.go:122] "RemoveContainer" containerID="5d012df886c8cacc3bef147272930be8845303c1f43d298a42cc8861251c0572" Nov 02 20:00:06.133496 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.131317 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6_odh-ai-gateway-infra(a96626ff-f0c1-42d2-a7ba-21d659e5313a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" Nov 02 20:00:06.133496 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.131489 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerDied","Data":"5d012df886c8cacc3bef147272930be8845303c1f43d298a42cc8861251c0572"} Nov 02 20:00:06.140880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.140395 4966 scope.go:122] "RemoveContainer" containerID="dcc832b3b2cc5a68bfa4136aec8fda4ace641a92cf614a02825934aff8bf0f6e" Nov 02 20:00:06.140880 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.140768 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h_odh-ai-gateway-infra(058383f3-e72a-4f34-b231-0acbde4b855d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" Nov 02 20:00:06.166908 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.166425 4966 scope.go:122] "RemoveContainer" containerID="70d1b01980afd22c37a80f3eb1d55c2bbda79455e771a707849675ab0d7f19cb" Nov 02 20:00:06.172349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.171063 4966 generic.go:356] "Generic (PLEG): container finished" podID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerID="ca800089ab4091d5d743b45ebfcada7bc07e7b1540d39ed784c42dc775d241d6" exitCode=60 Nov 02 20:00:06.172349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.171914 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" event={"ID":"5b514b7f-b2f8-4dc2-82cc-285eedff1071","Type":"ContainerDied","Data":"ca800089ab4091d5d743b45ebfcada7bc07e7b1540d39ed784c42dc775d241d6"} Nov 02 20:00:06.172533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.172458 4966 scope.go:122] "RemoveContainer" containerID="ca800089ab4091d5d743b45ebfcada7bc07e7b1540d39ed784c42dc775d241d6" Nov 02 20:00:06.172887 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.172853 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8_odh-ai-gateway-infra(5b514b7f-b2f8-4dc2-82cc-285eedff1071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" Nov 02 20:00:06.175630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.175599 4966 generic.go:356] "Generic (PLEG): container finished" podID="33e43a40-2677-4be6-9327-2e3424252de1" containerID="d041e0f84d4378fb9936107f4ba814039f5ac68ac538202a640656ed704c1cb0" exitCode=60 Nov 02 20:00:06.176152 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.176132 4966 scope.go:122] "RemoveContainer" containerID="d041e0f84d4378fb9936107f4ba814039f5ac68ac538202a640656ed704c1cb0" Nov 02 20:00:06.176682 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.176648 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf_odh-ai-gateway-infra(33e43a40-2677-4be6-9327-2e3424252de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" podUID="33e43a40-2677-4be6-9327-2e3424252de1" Nov 02 20:00:06.182956 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.182841 4966 generic.go:356] "Generic (PLEG): container finished" podID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerID="85b4d96d4b406f12bc467ef9484c8766de80d73dcfe0954c81425e995f14554f" exitCode=60 Nov 02 20:00:06.183290 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.183272 4966 scope.go:122] "RemoveContainer" containerID="85b4d96d4b406f12bc467ef9484c8766de80d73dcfe0954c81425e995f14554f" Nov 02 20:00:06.183604 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.183582 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf_odh-ai-gateway-infra(bc80a18b-04c3-44d6-809a-230cbde7d0db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" Nov 02 20:00:06.186377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.185953 4966 scope.go:122] "RemoveContainer" containerID="677821b2fe76c633384222ba611bb1b9535a0bb363124988b064abbc227f9c47" Nov 02 20:00:06.186377 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.186293 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4_odh-ai-gateway-infra(c0d12a6a-61da-4204-b826-75da5bb34463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" Nov 02 20:00:06.193288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.192775 4966 generic.go:356] "Generic (PLEG): container finished" podID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerID="3e22992da5e5211322e1fae1717bedb7ed0dfb2d73b863b38414b43bdc30366c" exitCode=60 Nov 02 20:00:06.193288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.193133 4966 scope.go:122] "RemoveContainer" containerID="3e22992da5e5211322e1fae1717bedb7ed0dfb2d73b863b38414b43bdc30366c" Nov 02 20:00:06.197411 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:06.197376 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss_odh-ai-gateway-infra(149c68c2-f8a0-4d24-8e66-e2abcfc3aacf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" Nov 02 20:00:06.199873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.199858 4966 scope.go:122] "RemoveContainer" containerID="17437a69cffb4a9247c3b82a0a0aa34f89fdfa67eccbf99ec90c7460b517387e" Nov 02 20:00:06.200765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.200738 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" event={"ID":"33e43a40-2677-4be6-9327-2e3424252de1","Type":"ContainerDied","Data":"d041e0f84d4378fb9936107f4ba814039f5ac68ac538202a640656ed704c1cb0"} Nov 02 20:00:06.200824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.200799 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerDied","Data":"85b4d96d4b406f12bc467ef9484c8766de80d73dcfe0954c81425e995f14554f"} Nov 02 20:00:06.200910 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.200828 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerDied","Data":"3e22992da5e5211322e1fae1717bedb7ed0dfb2d73b863b38414b43bdc30366c"} Nov 02 20:00:06.225553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.225525 4966 scope.go:122] "RemoveContainer" containerID="5b9e08528bd5df47bec0db82cf25d43ffa4409eddafb85fe4e5bd90080698926" Nov 02 20:00:06.249197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.249147 4966 scope.go:122] "RemoveContainer" containerID="761fda188f40d6e5bc6183a9d3e24a82d15f5e70e0bcc340892822d0208159b0" Nov 02 20:00:06.284659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.284543 4966 scope.go:122] "RemoveContainer" containerID="1466d3ea0f66dbfe798e9b813363eabd713a46e6f48a38983a9d29bbd8c2b3e3" Nov 02 20:00:06.432954 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:06.432926 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029571de_57d3_4dca_91f2_22fd2df6a513.slice/crio-682c03ee450d99647b0b667f296a1c84c33daaf11c30b75b106de15a5c7411f3 WatchSource:0}: Error finding container 682c03ee450d99647b0b667f296a1c84c33daaf11c30b75b106de15a5c7411f3: Status 404 returned error can't find the container with id 682c03ee450d99647b0b667f296a1c84c33daaf11c30b75b106de15a5c7411f3 Nov 02 20:00:06.433123 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.432999 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl"] Nov 02 20:00:06.539445 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:06.539414 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5c6025_a0af_455b_80b6_b38225915f4c.slice/crio-27afa3ee11eb0637f683b4e74541807b26521fef742fd417625b733048b9d67c WatchSource:0}: Error finding container 27afa3ee11eb0637f683b4e74541807b26521fef742fd417625b733048b9d67c: Status 404 returned error can't find the container with id 27afa3ee11eb0637f683b4e74541807b26521fef742fd417625b733048b9d67c Nov 02 20:00:06.540797 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:06.540775 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82"] Nov 02 20:00:07.217781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.217733 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" event={"ID":"9f5c6025-a0af-455b-80b6-b38225915f4c","Type":"ContainerStarted","Data":"9f0fa0f11640727bd6fd861a968c6e3ac25e68fa1be7fbdce0fc8f7f1fa23e64"} Nov 02 20:00:07.218447 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.217834 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" event={"ID":"9f5c6025-a0af-455b-80b6-b38225915f4c","Type":"ContainerStarted","Data":"27afa3ee11eb0637f683b4e74541807b26521fef742fd417625b733048b9d67c"} Nov 02 20:00:07.218447 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.217866 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:07.220129 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.220104 4966 scope.go:122] "RemoveContainer" containerID="4917f26cda6a06048342c3a0669b244b9df8277f0e9480f524e5506932ede966" Nov 02 20:00:07.220501 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:07.220476 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl_odh-ai-gateway-infra(a8a43c6d-a71c-4a48-a5b8-a16c10476d86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" Nov 02 20:00:07.223692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.223670 4966 scope.go:122] "RemoveContainer" containerID="5d012df886c8cacc3bef147272930be8845303c1f43d298a42cc8861251c0572" Nov 02 20:00:07.225572 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:07.225529 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6_odh-ai-gateway-infra(a96626ff-f0c1-42d2-a7ba-21d659e5313a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" Nov 02 20:00:07.228639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.228609 4966 scope.go:122] "RemoveContainer" containerID="ca800089ab4091d5d743b45ebfcada7bc07e7b1540d39ed784c42dc775d241d6" Nov 02 20:00:07.228953 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:07.228926 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8_odh-ai-gateway-infra(5b514b7f-b2f8-4dc2-82cc-285eedff1071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" Nov 02 20:00:07.230674 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.230654 4966 scope.go:122] "RemoveContainer" containerID="d041e0f84d4378fb9936107f4ba814039f5ac68ac538202a640656ed704c1cb0" Nov 02 20:00:07.230906 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:07.230887 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf_odh-ai-gateway-infra(33e43a40-2677-4be6-9327-2e3424252de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" podUID="33e43a40-2677-4be6-9327-2e3424252de1" Nov 02 20:00:07.232021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.231999 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" event={"ID":"029571de-57d3-4dca-91f2-22fd2df6a513","Type":"ContainerStarted","Data":"34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a"} Nov 02 20:00:07.232089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.232045 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" event={"ID":"029571de-57d3-4dca-91f2-22fd2df6a513","Type":"ContainerStarted","Data":"682c03ee450d99647b0b667f296a1c84c33daaf11c30b75b106de15a5c7411f3"} Nov 02 20:00:07.232089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.232073 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:07.233716 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.233698 4966 scope.go:122] "RemoveContainer" containerID="85b4d96d4b406f12bc467ef9484c8766de80d73dcfe0954c81425e995f14554f" Nov 02 20:00:07.233944 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:07.233923 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf_odh-ai-gateway-infra(bc80a18b-04c3-44d6-809a-230cbde7d0db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" Nov 02 20:00:07.235550 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:07.235533 4966 scope.go:122] "RemoveContainer" containerID="3e22992da5e5211322e1fae1717bedb7ed0dfb2d73b863b38414b43bdc30366c" Nov 02 20:00:07.235748 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:07.235726 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss_odh-ai-gateway-infra(149c68c2-f8a0-4d24-8e66-e2abcfc3aacf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" Nov 02 20:00:08.300313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:08.300242 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" podStartSLOduration=3.300222701 podStartE2EDuration="3.300222701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:08.294911237 +0000 UTC m=+1914.243567076" watchObservedRunningTime="2026-11-02 20:00:08.300222701 +0000 UTC m=+1914.248878543" Nov 02 20:00:08.821880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:08.821847 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 20:00:08.822080 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:08.821938 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 20:00:08.957854 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:08.957801 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" podStartSLOduration=3.9577875860000002 podStartE2EDuration="3.957787586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:08.95459824 +0000 UTC m=+1914.903254079" watchObservedRunningTime="2026-11-02 20:00:08.957787586 +0000 UTC m=+1914.906443422" Nov 02 20:00:10.139685 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.139653 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 20:00:10.263712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.263676 4966 generic.go:356] "Generic (PLEG): container finished" podID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerID="07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36" exitCode=0 Nov 02 20:00:10.263880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.263726 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" Nov 02 20:00:10.263880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.263739 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" event={"ID":"59f5dae8-66ca-497f-aec4-220ffc3ac4b8","Type":"ContainerDied","Data":"07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36"} Nov 02 20:00:10.263880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.263790 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n" event={"ID":"59f5dae8-66ca-497f-aec4-220ffc3ac4b8","Type":"ContainerDied","Data":"1d25f6ae7218c0aa1157818039e13d72fc670a9a25102bcdb48c3a95eb3ee022"} Nov 02 20:00:10.263880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.263820 4966 scope.go:122] "RemoveContainer" containerID="07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36" Nov 02 20:00:10.279401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.279373 4966 scope.go:122] "RemoveContainer" containerID="07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36" Nov 02 20:00:10.279752 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:10.279714 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36\": container with ID starting with 07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36 not found: ID does not exist" containerID="07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36" Nov 02 20:00:10.279890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.279759 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36"} err="failed to get container status \"07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36\": rpc error: code = NotFound desc = could not find container \"07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36\": container with ID starting with 07587733c3c286f932708ff1e70a23b93fe544293909842888600800a540bf36 not found: ID does not exist" Nov 02 20:00:10.291387 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291353 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-socket\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291509 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291410 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-podinfo\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291524 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-certs\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291581 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-token\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-token\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291623 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-socket" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:10.291727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291634 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-kube-api-access-kbxtr\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-kube-api-access-kbxtr\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291775 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istiod-ca-cert\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291894 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291873 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-envoy\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291908 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-credential-socket\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.291952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291776 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-certs" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:10.292068 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.291990 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-data\") pod \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\" (UID: \"59f5dae8-66ca-497f-aec4-220ffc3ac4b8\") " Nov 02 20:00:10.292068 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.292060 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istiod-ca-cert" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:10.292347 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.292286 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-credential-socket" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:10.292539 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.292509 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-data" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:10.293021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.292995 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.293117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.293034 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.293117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.293052 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.293117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.293066 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.293117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.293080 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.294255 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.294233 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-podinfo" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:10.294608 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.294589 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-kube-api-access-kbxtr" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "kube-api-access-kbxtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:10.295012 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.294997 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-token" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:10.295090 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.295070 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-envoy" pod "59f5dae8-66ca-497f-aec4-220ffc3ac4b8" (UID: "59f5dae8-66ca-497f-aec4-220ffc3ac4b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:10.396512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.396425 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.396512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.396485 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.396512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.396503 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.396904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.396518 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxtr\" (UniqueName: \"kubernetes.io/projected/59f5dae8-66ca-497f-aec4-220ffc3ac4b8-kube-api-access-kbxtr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:10.711822 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.711707 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n"] Nov 02 20:00:10.717828 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:10.717494 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-qf92n"] Nov 02 20:00:12.189437 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:12.189400 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" path="/var/lib/kubelet/pods/59f5dae8-66ca-497f-aec4-220ffc3ac4b8/volumes" Nov 02 20:00:14.758730 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:14.758686 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:00:15.180769 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:15.178448 4966 scope.go:122] "RemoveContainer" containerID="a31777a88c64f43f4d03dcb939e2c30df0ff8c546b627bba9b3aea3aec33684b" Nov 02 20:00:16.177012 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:16.176974 4966 scope.go:122] "RemoveContainer" containerID="866c1bb595e86b29d80ad034011231aaca594a570f8057c6f5b360117f9ce497" Nov 02 20:00:16.177385 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:16.177092 4966 scope.go:122] "RemoveContainer" containerID="8212002fe57845ae968554db4bf2c8cb55fbd6fc24fb389c53cb3d34167c8f8b" Nov 02 20:00:16.352090 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:16.352056 4966 generic.go:356] "Generic (PLEG): container finished" podID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerID="d374b6945c7c166a4d82067e64cdba3c1add19b56399e548e7a40d624b81b19e" exitCode=60 Nov 02 20:00:16.352280 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:16.352131 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" event={"ID":"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4","Type":"ContainerDied","Data":"d374b6945c7c166a4d82067e64cdba3c1add19b56399e548e7a40d624b81b19e"} Nov 02 20:00:16.352280 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:16.352199 4966 scope.go:122] "RemoveContainer" containerID="a31777a88c64f43f4d03dcb939e2c30df0ff8c546b627bba9b3aea3aec33684b" Nov 02 20:00:16.352704 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:16.352685 4966 scope.go:122] "RemoveContainer" containerID="d374b6945c7c166a4d82067e64cdba3c1add19b56399e548e7a40d624b81b19e" Nov 02 20:00:16.353147 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:16.353114 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt_odh-ai-gateway-infra(03e09a4f-2cc9-4044-8ddd-18eedfd04ff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" Nov 02 20:00:17.175849 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.175819 4966 scope.go:122] "RemoveContainer" containerID="5d012df886c8cacc3bef147272930be8845303c1f43d298a42cc8861251c0572" Nov 02 20:00:17.176013 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.175943 4966 scope.go:122] "RemoveContainer" containerID="727b2d3ccd8e8944a8217fac7b0a2d7f2ce2e104cac4a0687b22ee00728e90d7" Nov 02 20:00:17.176214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.176190 4966 scope.go:122] "RemoveContainer" containerID="3e22992da5e5211322e1fae1717bedb7ed0dfb2d73b863b38414b43bdc30366c" Nov 02 20:00:17.368011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.367976 4966 generic.go:356] "Generic (PLEG): container finished" podID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerID="aca4d471b4e6c46d46f53cc2fc685e68f1538ddec794f622602180545caffbbc" exitCode=60 Nov 02 20:00:17.368515 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.368053 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" event={"ID":"a78d71ed-3361-4b0e-98fc-c3cb9f174a02","Type":"ContainerDied","Data":"aca4d471b4e6c46d46f53cc2fc685e68f1538ddec794f622602180545caffbbc"} Nov 02 20:00:17.368776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.368158 4966 scope.go:122] "RemoveContainer" containerID="866c1bb595e86b29d80ad034011231aaca594a570f8057c6f5b360117f9ce497" Nov 02 20:00:17.368885 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.368817 4966 scope.go:122] "RemoveContainer" containerID="aca4d471b4e6c46d46f53cc2fc685e68f1538ddec794f622602180545caffbbc" Nov 02 20:00:17.369264 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:17.369238 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d_odh-ai-gateway-infra(a78d71ed-3361-4b0e-98fc-c3cb9f174a02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" Nov 02 20:00:17.372397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.372372 4966 generic.go:356] "Generic (PLEG): container finished" podID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerID="72aebb1ca51702b7065c4960d4427f6cadb43c683710e47bce61bb0a4d0e0742" exitCode=60 Nov 02 20:00:17.372507 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.372440 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" event={"ID":"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e","Type":"ContainerDied","Data":"72aebb1ca51702b7065c4960d4427f6cadb43c683710e47bce61bb0a4d0e0742"} Nov 02 20:00:17.372893 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.372878 4966 scope.go:122] "RemoveContainer" containerID="72aebb1ca51702b7065c4960d4427f6cadb43c683710e47bce61bb0a4d0e0742" Nov 02 20:00:17.373277 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:17.373255 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7_odh-ai-gateway-infra(e015108d-dcbb-4fec-a3a9-3a7fd0f8178e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" Nov 02 20:00:17.384945 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.384924 4966 scope.go:122] "RemoveContainer" containerID="8212002fe57845ae968554db4bf2c8cb55fbd6fc24fb389c53cb3d34167c8f8b" Nov 02 20:00:17.472244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.472210 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt"] Nov 02 20:00:17.730607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.730582 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:17.808445 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.808419 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-maas-api-service-ca\") pod \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " Nov 02 20:00:17.808574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.808522 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-kube-api-access-lnh9s\" (UniqueName: \"kubernetes.io/projected/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-kube-api-access-lnh9s\") pod \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\" (UID: \"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4\") " Nov 02 20:00:17.808785 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.808763 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-maas-api-service-ca" pod "03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" (UID: "03e09a4f-2cc9-4044-8ddd-18eedfd04ff4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:17.809011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.808998 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:17.810764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.810740 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-kube-api-access-lnh9s" pod "03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" (UID: "03e09a4f-2cc9-4044-8ddd-18eedfd04ff4"). InnerVolumeSpecName "kube-api-access-lnh9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:17.910229 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:17.910150 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnh9s\" (UniqueName: \"kubernetes.io/projected/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4-kube-api-access-lnh9s\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:18.017374 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.017341 4966 scope.go:122] "RemoveContainer" containerID="671bf852f9daf7be7f5dd3a039ccb9dedbde4f735a2d58068e7e27c372735035" Nov 02 20:00:18.028730 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.028703 4966 scope.go:122] "RemoveContainer" containerID="0e7dfa90c5492f05d8073908df06245a64b55f9d1897863470f4a711d1ae3116" Nov 02 20:00:18.176266 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.176175 4966 scope.go:122] "RemoveContainer" containerID="dcc832b3b2cc5a68bfa4136aec8fda4ace641a92cf614a02825934aff8bf0f6e" Nov 02 20:00:18.176479 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.176406 4966 scope.go:122] "RemoveContainer" containerID="0471fde7494e8414af417fa95f89ca95bd4b91484bf4f0d5820e1925dd1a4e88" Nov 02 20:00:18.176552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.176520 4966 scope.go:122] "RemoveContainer" containerID="85b4d96d4b406f12bc467ef9484c8766de80d73dcfe0954c81425e995f14554f" Nov 02 20:00:18.176947 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.176919 4966 scope.go:122] "RemoveContainer" containerID="4917f26cda6a06048342c3a0669b244b9df8277f0e9480f524e5506932ede966" Nov 02 20:00:18.250971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.250944 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:00:18.251938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.251910 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:00:18.397661 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.397619 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" event={"ID":"03e09a4f-2cc9-4044-8ddd-18eedfd04ff4","Type":"ContainerDied","Data":"95ac553b5499a2069cb4dd12b6c4023bacd0874ef1b45ee2a02f3da7d046120d"} Nov 02 20:00:18.398171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.397687 4966 scope.go:122] "RemoveContainer" containerID="d374b6945c7c166a4d82067e64cdba3c1add19b56399e548e7a40d624b81b19e" Nov 02 20:00:18.398171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.397819 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt" Nov 02 20:00:18.418066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.415701 4966 generic.go:356] "Generic (PLEG): container finished" podID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerID="04551e3477d17514bed32f905c6c1a02fba1b2275456793d06dd7420ccb5b5e2" exitCode=60 Nov 02 20:00:18.418066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.415774 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" event={"ID":"789d89be-2cd9-481f-a5ed-2590ff0c163c","Type":"ContainerDied","Data":"04551e3477d17514bed32f905c6c1a02fba1b2275456793d06dd7420ccb5b5e2"} Nov 02 20:00:18.418066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.416639 4966 scope.go:122] "RemoveContainer" containerID="04551e3477d17514bed32f905c6c1a02fba1b2275456793d06dd7420ccb5b5e2" Nov 02 20:00:18.418066 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:18.417176 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b_odh-ai-gateway-infra(789d89be-2cd9-481f-a5ed-2590ff0c163c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" Nov 02 20:00:18.419245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.419216 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerStarted","Data":"8e3559f4f80de658ce6d83ab4f452ff282736ca4568088a21bafc1e1b236f599"} Nov 02 20:00:18.422248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.422213 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerStarted","Data":"0ca6eea3be478a2778b3daf621944af163818ff2e814fda9ca9c71f03b568d07"} Nov 02 20:00:18.493948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.493669 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d"] Nov 02 20:00:18.614658 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.614624 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7"] Nov 02 20:00:18.633592 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.633567 4966 scope.go:122] "RemoveContainer" containerID="727b2d3ccd8e8944a8217fac7b0a2d7f2ce2e104cac4a0687b22ee00728e90d7" Nov 02 20:00:18.704685 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.704621 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:18.832980 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.832943 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-maas-api-service-ca\") pod \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " Nov 02 20:00:18.833175 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.833036 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-kube-api-access-lc9c7\" (UniqueName: \"kubernetes.io/projected/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-kube-api-access-lc9c7\") pod \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\" (UID: \"a78d71ed-3361-4b0e-98fc-c3cb9f174a02\") " Nov 02 20:00:18.833583 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.833327 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-maas-api-service-ca" pod "a78d71ed-3361-4b0e-98fc-c3cb9f174a02" (UID: "a78d71ed-3361-4b0e-98fc-c3cb9f174a02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:18.833800 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.833768 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:18.836049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.836019 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-kube-api-access-lc9c7" pod "a78d71ed-3361-4b0e-98fc-c3cb9f174a02" (UID: "a78d71ed-3361-4b0e-98fc-c3cb9f174a02"). InnerVolumeSpecName "kube-api-access-lc9c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:18.840533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.840224 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:18.934551 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.934517 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-kube-api-access-r4njd\" (UniqueName: \"kubernetes.io/projected/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-kube-api-access-r4njd\") pod \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " Nov 02 20:00:18.934723 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.934607 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-maas-api-service-ca\") pod \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\" (UID: \"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e\") " Nov 02 20:00:18.934999 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.934971 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-maas-api-service-ca" pod "e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" (UID: "e015108d-dcbb-4fec-a3a9-3a7fd0f8178e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:18.935356 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.935338 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9c7\" (UniqueName: \"kubernetes.io/projected/a78d71ed-3361-4b0e-98fc-c3cb9f174a02-kube-api-access-lc9c7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:18.935424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.935358 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:18.936764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:18.936740 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-kube-api-access-r4njd" pod "e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" (UID: "e015108d-dcbb-4fec-a3a9-3a7fd0f8178e"). InnerVolumeSpecName "kube-api-access-r4njd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:19.036597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.036560 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4njd\" (UniqueName: \"kubernetes.io/projected/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e-kube-api-access-r4njd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:19.161560 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.161514 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt"] Nov 02 20:00:19.171277 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.171229 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0b7eed-29894160-97slt"] Nov 02 20:00:19.176677 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.176648 4966 scope.go:122] "RemoveContainer" containerID="677821b2fe76c633384222ba611bb1b9535a0bb363124988b064abbc227f9c47" Nov 02 20:00:19.177016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.176998 4966 scope.go:122] "RemoveContainer" containerID="d041e0f84d4378fb9936107f4ba814039f5ac68ac538202a640656ed704c1cb0" Nov 02 20:00:19.436833 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.436753 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" Nov 02 20:00:19.436833 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.436781 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7" event={"ID":"e015108d-dcbb-4fec-a3a9-3a7fd0f8178e","Type":"ContainerDied","Data":"ce028bc164d4f937a69ee2e173a483b7bbcacec59f83ced34a61fb4d243c61de"} Nov 02 20:00:19.437371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.436840 4966 scope.go:122] "RemoveContainer" containerID="72aebb1ca51702b7065c4960d4427f6cadb43c683710e47bce61bb0a4d0e0742" Nov 02 20:00:19.448818 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.448287 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" event={"ID":"a78d71ed-3361-4b0e-98fc-c3cb9f174a02","Type":"ContainerDied","Data":"c52a07bd0c51a04cd2be114298d3cb34759988977a43b5b270dc9dc9f34231e6"} Nov 02 20:00:19.448818 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.448466 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d" Nov 02 20:00:19.483763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.479404 4966 scope.go:122] "RemoveContainer" containerID="aca4d471b4e6c46d46f53cc2fc685e68f1538ddec794f622602180545caffbbc" Nov 02 20:00:19.483763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.483498 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerStarted","Data":"a49cfb5b0834fb8f8dfc25ea720ec9b2851979328c5eea52397b1b24808ddd05"} Nov 02 20:00:19.501742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.500250 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerStarted","Data":"82869440cb6a4c3560618a1fbf0d13add2988bcbc21424cccbcb9d544163cd86"} Nov 02 20:00:19.506613 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.504446 4966 generic.go:356] "Generic (PLEG): container finished" podID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerID="8e3559f4f80de658ce6d83ab4f452ff282736ca4568088a21bafc1e1b236f599" exitCode=60 Nov 02 20:00:19.506613 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.504565 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerDied","Data":"8e3559f4f80de658ce6d83ab4f452ff282736ca4568088a21bafc1e1b236f599"} Nov 02 20:00:19.506613 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.505120 4966 scope.go:122] "RemoveContainer" containerID="8e3559f4f80de658ce6d83ab4f452ff282736ca4568088a21bafc1e1b236f599" Nov 02 20:00:19.506613 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:19.505708 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6_odh-ai-gateway-infra(a96626ff-f0c1-42d2-a7ba-21d659e5313a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" Nov 02 20:00:19.510240 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.510200 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" event={"ID":"058383f3-e72a-4f34-b231-0acbde4b855d","Type":"ContainerStarted","Data":"e3eb3c62b0f3b124f727c177f6d713a3b3d84f140cf90ed491aeebc93428a125"} Nov 02 20:00:19.512107 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.512086 4966 scope.go:122] "RemoveContainer" containerID="5d012df886c8cacc3bef147272930be8845303c1f43d298a42cc8861251c0572" Nov 02 20:00:19.519047 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.517698 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerStarted","Data":"d9d46ee91623c0ffb65fcec86b0178ea197b1dc88a65c35723876aabe55d273f"} Nov 02 20:00:19.520410 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.520384 4966 generic.go:356] "Generic (PLEG): container finished" podID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerID="0ca6eea3be478a2778b3daf621944af163818ff2e814fda9ca9c71f03b568d07" exitCode=60 Nov 02 20:00:19.520489 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.520420 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerDied","Data":"0ca6eea3be478a2778b3daf621944af163818ff2e814fda9ca9c71f03b568d07"} Nov 02 20:00:19.520844 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.520826 4966 scope.go:122] "RemoveContainer" containerID="0ca6eea3be478a2778b3daf621944af163818ff2e814fda9ca9c71f03b568d07" Nov 02 20:00:19.521211 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:19.521179 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss_odh-ai-gateway-infra(149c68c2-f8a0-4d24-8e66-e2abcfc3aacf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" Nov 02 20:00:19.531711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.531688 4966 scope.go:122] "RemoveContainer" containerID="3e22992da5e5211322e1fae1717bedb7ed0dfb2d73b863b38414b43bdc30366c" Nov 02 20:00:19.592695 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.592635 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6"] Nov 02 20:00:19.739241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.739204 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b"] Nov 02 20:00:19.892845 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.892817 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss"] Nov 02 20:00:19.991918 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.991881 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7"] Nov 02 20:00:19.997946 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:19.997919 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894160-w6sr7"] Nov 02 20:00:20.034998 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.034974 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:20.189037 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.189000 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" path="/var/lib/kubelet/pods/03e09a4f-2cc9-4044-8ddd-18eedfd04ff4/volumes" Nov 02 20:00:20.189589 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.189573 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" path="/var/lib/kubelet/pods/e015108d-dcbb-4fec-a3a9-3a7fd0f8178e/volumes" Nov 02 20:00:20.190585 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.190569 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789d89be-2cd9-481f-a5ed-2590ff0c163c-kube-api-access-cknhx\" (UniqueName: \"kubernetes.io/projected/789d89be-2cd9-481f-a5ed-2590ff0c163c-kube-api-access-cknhx\") pod \"789d89be-2cd9-481f-a5ed-2590ff0c163c\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " Nov 02 20:00:20.190663 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.190650 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789d89be-2cd9-481f-a5ed-2590ff0c163c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789d89be-2cd9-481f-a5ed-2590ff0c163c-maas-api-service-ca\") pod \"789d89be-2cd9-481f-a5ed-2590ff0c163c\" (UID: \"789d89be-2cd9-481f-a5ed-2590ff0c163c\") " Nov 02 20:00:20.190974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.190953 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789d89be-2cd9-481f-a5ed-2590ff0c163c-maas-api-service-ca" pod "789d89be-2cd9-481f-a5ed-2590ff0c163c" (UID: "789d89be-2cd9-481f-a5ed-2590ff0c163c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:20.191215 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.191197 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789d89be-2cd9-481f-a5ed-2590ff0c163c-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:20.192361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.192337 4966 scope.go:122] "RemoveContainer" containerID="ca800089ab4091d5d743b45ebfcada7bc07e7b1540d39ed784c42dc775d241d6" Nov 02 20:00:20.192779 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.192755 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789d89be-2cd9-481f-a5ed-2590ff0c163c-kube-api-access-cknhx" pod "789d89be-2cd9-481f-a5ed-2590ff0c163c" (UID: "789d89be-2cd9-481f-a5ed-2590ff0c163c"). InnerVolumeSpecName "kube-api-access-cknhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:20.198891 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.198865 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d"] Nov 02 20:00:20.203689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.203667 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894160-z955d"] Nov 02 20:00:20.293073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.292982 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknhx\" (UniqueName: \"kubernetes.io/projected/789d89be-2cd9-481f-a5ed-2590ff0c163c-kube-api-access-cknhx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:20.532717 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.532687 4966 generic.go:356] "Generic (PLEG): container finished" podID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerID="d9d46ee91623c0ffb65fcec86b0178ea197b1dc88a65c35723876aabe55d273f" exitCode=60 Nov 02 20:00:20.533204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.532761 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerDied","Data":"d9d46ee91623c0ffb65fcec86b0178ea197b1dc88a65c35723876aabe55d273f"} Nov 02 20:00:20.533204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.532836 4966 scope.go:122] "RemoveContainer" containerID="85b4d96d4b406f12bc467ef9484c8766de80d73dcfe0954c81425e995f14554f" Nov 02 20:00:20.534622 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.534258 4966 scope.go:122] "RemoveContainer" containerID="d9d46ee91623c0ffb65fcec86b0178ea197b1dc88a65c35723876aabe55d273f" Nov 02 20:00:20.534739 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:20.534716 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf_odh-ai-gateway-infra(bc80a18b-04c3-44d6-809a-230cbde7d0db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" Nov 02 20:00:20.536416 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.536390 4966 generic.go:356] "Generic (PLEG): container finished" podID="c0d12a6a-61da-4204-b826-75da5bb34463" containerID="4f88140e0789b3c66bdd43df7507106928dce41aeab67b020f2aed6eb3d04f50" exitCode=60 Nov 02 20:00:20.536527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.536425 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" event={"ID":"c0d12a6a-61da-4204-b826-75da5bb34463","Type":"ContainerDied","Data":"4f88140e0789b3c66bdd43df7507106928dce41aeab67b020f2aed6eb3d04f50"} Nov 02 20:00:20.536846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.536828 4966 scope.go:122] "RemoveContainer" containerID="4f88140e0789b3c66bdd43df7507106928dce41aeab67b020f2aed6eb3d04f50" Nov 02 20:00:20.537191 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:20.537166 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4_odh-ai-gateway-infra(c0d12a6a-61da-4204-b826-75da5bb34463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" Nov 02 20:00:20.544659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.544598 4966 generic.go:356] "Generic (PLEG): container finished" podID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerID="a49cfb5b0834fb8f8dfc25ea720ec9b2851979328c5eea52397b1b24808ddd05" exitCode=60 Nov 02 20:00:20.544753 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.544699 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerDied","Data":"a49cfb5b0834fb8f8dfc25ea720ec9b2851979328c5eea52397b1b24808ddd05"} Nov 02 20:00:20.545697 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.545670 4966 scope.go:122] "RemoveContainer" containerID="a49cfb5b0834fb8f8dfc25ea720ec9b2851979328c5eea52397b1b24808ddd05" Nov 02 20:00:20.547142 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:20.546049 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl_odh-ai-gateway-infra(a8a43c6d-a71c-4a48-a5b8-a16c10476d86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" Nov 02 20:00:20.547746 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.547725 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" Nov 02 20:00:20.548007 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.547987 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b" event={"ID":"789d89be-2cd9-481f-a5ed-2590ff0c163c","Type":"ContainerDied","Data":"4365057c2d916cb0c9487fcc943070e08c33c8ce97745b96e8b4e329c4b39fb0"} Nov 02 20:00:20.555644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.555623 4966 scope.go:122] "RemoveContainer" containerID="677821b2fe76c633384222ba611bb1b9535a0bb363124988b064abbc227f9c47" Nov 02 20:00:20.562278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.561718 4966 generic.go:356] "Generic (PLEG): container finished" podID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerID="82869440cb6a4c3560618a1fbf0d13add2988bcbc21424cccbcb9d544163cd86" exitCode=60 Nov 02 20:00:20.562278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.561835 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerDied","Data":"82869440cb6a4c3560618a1fbf0d13add2988bcbc21424cccbcb9d544163cd86"} Nov 02 20:00:20.562278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.562155 4966 scope.go:122] "RemoveContainer" containerID="82869440cb6a4c3560618a1fbf0d13add2988bcbc21424cccbcb9d544163cd86" Nov 02 20:00:20.562506 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:20.562469 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6_odh-ai-gateway-infra(42e5bb7d-7b64-4088-9ffd-7ce1ba735894)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" Nov 02 20:00:20.566961 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.566940 4966 generic.go:356] "Generic (PLEG): container finished" podID="058383f3-e72a-4f34-b231-0acbde4b855d" containerID="e3eb3c62b0f3b124f727c177f6d713a3b3d84f140cf90ed491aeebc93428a125" exitCode=60 Nov 02 20:00:20.567081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.567013 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" event={"ID":"058383f3-e72a-4f34-b231-0acbde4b855d","Type":"ContainerDied","Data":"e3eb3c62b0f3b124f727c177f6d713a3b3d84f140cf90ed491aeebc93428a125"} Nov 02 20:00:20.569391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.569370 4966 scope.go:122] "RemoveContainer" containerID="e3eb3c62b0f3b124f727c177f6d713a3b3d84f140cf90ed491aeebc93428a125" Nov 02 20:00:20.570431 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.569710 4966 generic.go:356] "Generic (PLEG): container finished" podID="33e43a40-2677-4be6-9327-2e3424252de1" containerID="655f7f45c110556817c9cde468d8d99f068f296cd9f71c32e95621ec9e2a9f22" exitCode=60 Nov 02 20:00:20.570431 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.569768 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" event={"ID":"33e43a40-2677-4be6-9327-2e3424252de1","Type":"ContainerDied","Data":"655f7f45c110556817c9cde468d8d99f068f296cd9f71c32e95621ec9e2a9f22"} Nov 02 20:00:20.570431 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.570071 4966 scope.go:122] "RemoveContainer" containerID="655f7f45c110556817c9cde468d8d99f068f296cd9f71c32e95621ec9e2a9f22" Nov 02 20:00:20.570431 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:20.570377 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf_odh-ai-gateway-infra(33e43a40-2677-4be6-9327-2e3424252de1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" podUID="33e43a40-2677-4be6-9327-2e3424252de1" Nov 02 20:00:20.570885 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:20.570864 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h_odh-ai-gateway-infra(058383f3-e72a-4f34-b231-0acbde4b855d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" Nov 02 20:00:20.583463 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.583430 4966 scope.go:122] "RemoveContainer" containerID="4917f26cda6a06048342c3a0669b244b9df8277f0e9480f524e5506932ede966" Nov 02 20:00:20.604465 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.604101 4966 scope.go:122] "RemoveContainer" containerID="04551e3477d17514bed32f905c6c1a02fba1b2275456793d06dd7420ccb5b5e2" Nov 02 20:00:20.616210 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.616182 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl"] Nov 02 20:00:20.622426 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.622399 4966 scope.go:122] "RemoveContainer" containerID="0471fde7494e8414af417fa95f89ca95bd4b91484bf4f0d5820e1925dd1a4e88" Nov 02 20:00:20.672018 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.671996 4966 scope.go:122] "RemoveContainer" containerID="dcc832b3b2cc5a68bfa4136aec8fda4ace641a92cf614a02825934aff8bf0f6e" Nov 02 20:00:20.709491 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.709469 4966 scope.go:122] "RemoveContainer" containerID="d041e0f84d4378fb9936107f4ba814039f5ac68ac538202a640656ed704c1cb0" Nov 02 20:00:20.785684 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.785643 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6"] Nov 02 20:00:20.792587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.792511 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:20.815771 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.815744 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:20.901673 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.901640 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96626ff-f0c1-42d2-a7ba-21d659e5313a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96626ff-f0c1-42d2-a7ba-21d659e5313a-maas-api-service-ca\") pod \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " Nov 02 20:00:20.901858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.901720 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-kube-api-access-flv9n\" (UniqueName: \"kubernetes.io/projected/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-kube-api-access-flv9n\") pod \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " Nov 02 20:00:20.901858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.901811 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96626ff-f0c1-42d2-a7ba-21d659e5313a-kube-api-access-7phb5\" (UniqueName: \"kubernetes.io/projected/a96626ff-f0c1-42d2-a7ba-21d659e5313a-kube-api-access-7phb5\") pod \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\" (UID: \"a96626ff-f0c1-42d2-a7ba-21d659e5313a\") " Nov 02 20:00:20.902075 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.901897 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-maas-api-service-ca\") pod \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\" (UID: \"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf\") " Nov 02 20:00:20.902188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.902163 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96626ff-f0c1-42d2-a7ba-21d659e5313a-maas-api-service-ca" pod "a96626ff-f0c1-42d2-a7ba-21d659e5313a" (UID: "a96626ff-f0c1-42d2-a7ba-21d659e5313a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:20.902269 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.902248 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-maas-api-service-ca" pod "149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" (UID: "149c68c2-f8a0-4d24-8e66-e2abcfc3aacf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:20.902604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.902589 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96626ff-f0c1-42d2-a7ba-21d659e5313a-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:20.902648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.902606 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:20.903867 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.903847 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-kube-api-access-flv9n" pod "149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" (UID: "149c68c2-f8a0-4d24-8e66-e2abcfc3aacf"). InnerVolumeSpecName "kube-api-access-flv9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:20.903908 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:20.903865 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96626ff-f0c1-42d2-a7ba-21d659e5313a-kube-api-access-7phb5" pod "a96626ff-f0c1-42d2-a7ba-21d659e5313a" (UID: "a96626ff-f0c1-42d2-a7ba-21d659e5313a"). InnerVolumeSpecName "kube-api-access-7phb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:21.004324 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.004255 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flv9n\" (UniqueName: \"kubernetes.io/projected/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf-kube-api-access-flv9n\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:21.004324 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.004293 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7phb5\" (UniqueName: \"kubernetes.io/projected/a96626ff-f0c1-42d2-a7ba-21d659e5313a-kube-api-access-7phb5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:21.109027 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.108929 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h"] Nov 02 20:00:21.194979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.194941 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b"] Nov 02 20:00:21.199842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.199812 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894160-qsv9b"] Nov 02 20:00:21.432515 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.432417 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf"] Nov 02 20:00:21.588783 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.588750 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" Nov 02 20:00:21.589671 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.588773 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss" event={"ID":"149c68c2-f8a0-4d24-8e66-e2abcfc3aacf","Type":"ContainerDied","Data":"b9cef6cf3cdb34c3148f74f0436dbe47f7a3bf1d2bf30744341dfe65006322d6"} Nov 02 20:00:21.589671 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.588864 4966 scope.go:122] "RemoveContainer" containerID="0ca6eea3be478a2778b3daf621944af163818ff2e814fda9ca9c71f03b568d07" Nov 02 20:00:21.597520 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.597475 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" event={"ID":"a96626ff-f0c1-42d2-a7ba-21d659e5313a","Type":"ContainerDied","Data":"da2bce5c7dbf75a88bc6e0ced30e3f6e5f556332dd05d387aa47066244b6472b"} Nov 02 20:00:21.597675 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.597627 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6" Nov 02 20:00:21.608728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.608683 4966 scope.go:122] "RemoveContainer" containerID="8e3559f4f80de658ce6d83ab4f452ff282736ca4568088a21bafc1e1b236f599" Nov 02 20:00:21.617066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.609725 4966 generic.go:356] "Generic (PLEG): container finished" podID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerID="481e4b32176818cf8132e10e79c4667f7e515664487129c3aeb86ce320cd67ac" exitCode=60 Nov 02 20:00:21.617066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.609855 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" event={"ID":"5b514b7f-b2f8-4dc2-82cc-285eedff1071","Type":"ContainerDied","Data":"481e4b32176818cf8132e10e79c4667f7e515664487129c3aeb86ce320cd67ac"} Nov 02 20:00:21.617066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.610431 4966 scope.go:122] "RemoveContainer" containerID="481e4b32176818cf8132e10e79c4667f7e515664487129c3aeb86ce320cd67ac" Nov 02 20:00:21.617066 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:21.610867 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8_odh-ai-gateway-infra(5b514b7f-b2f8-4dc2-82cc-285eedff1071)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" Nov 02 20:00:21.631204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.631175 4966 scope.go:122] "RemoveContainer" containerID="ca800089ab4091d5d743b45ebfcada7bc07e7b1540d39ed784c42dc775d241d6" Nov 02 20:00:21.892762 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.892732 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4"] Nov 02 20:00:21.933579 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.933341 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6"] Nov 02 20:00:21.941564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.941459 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-4fd0d5-29894160-75jn6"] Nov 02 20:00:21.962326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.962018 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:21.966786 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.966759 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:21.978650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:21.978591 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:22.025475 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.025443 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-kube-api-access-j7bvq\" (UniqueName: \"kubernetes.io/projected/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-kube-api-access-j7bvq\") pod \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " Nov 02 20:00:22.025662 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.025517 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-maas-api-service-ca\") pod \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\" (UID: \"42e5bb7d-7b64-4088-9ffd-7ce1ba735894\") " Nov 02 20:00:22.025662 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.025579 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-kube-api-access-5wk9f\" (UniqueName: \"kubernetes.io/projected/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-kube-api-access-5wk9f\") pod \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " Nov 02 20:00:22.025760 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.025694 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-maas-api-service-ca\") pod \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\" (UID: \"a8a43c6d-a71c-4a48-a5b8-a16c10476d86\") " Nov 02 20:00:22.026900 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.026867 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-maas-api-service-ca" pod "a8a43c6d-a71c-4a48-a5b8-a16c10476d86" (UID: "a8a43c6d-a71c-4a48-a5b8-a16c10476d86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:22.027538 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.027511 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-maas-api-service-ca" pod "42e5bb7d-7b64-4088-9ffd-7ce1ba735894" (UID: "42e5bb7d-7b64-4088-9ffd-7ce1ba735894"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:22.031058 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.031032 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-kube-api-access-j7bvq" pod "42e5bb7d-7b64-4088-9ffd-7ce1ba735894" (UID: "42e5bb7d-7b64-4088-9ffd-7ce1ba735894"). InnerVolumeSpecName "kube-api-access-j7bvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:22.031493 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.031466 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-kube-api-access-5wk9f" pod "a8a43c6d-a71c-4a48-a5b8-a16c10476d86" (UID: "a8a43c6d-a71c-4a48-a5b8-a16c10476d86"). InnerVolumeSpecName "kube-api-access-5wk9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:22.060562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.060532 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:22.117044 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.117021 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:22.127210 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.127188 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc80a18b-04c3-44d6-809a-230cbde7d0db-kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/bc80a18b-04c3-44d6-809a-230cbde7d0db-kube-api-access-bsd5f\") pod \"bc80a18b-04c3-44d6-809a-230cbde7d0db\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " Nov 02 20:00:22.127356 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.127292 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058383f3-e72a-4f34-b231-0acbde4b855d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058383f3-e72a-4f34-b231-0acbde4b855d-maas-api-service-ca\") pod \"058383f3-e72a-4f34-b231-0acbde4b855d\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " Nov 02 20:00:22.127515 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.127496 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058383f3-e72a-4f34-b231-0acbde4b855d-kube-api-access-4plr8\" (UniqueName: \"kubernetes.io/projected/058383f3-e72a-4f34-b231-0acbde4b855d-kube-api-access-4plr8\") pod \"058383f3-e72a-4f34-b231-0acbde4b855d\" (UID: \"058383f3-e72a-4f34-b231-0acbde4b855d\") " Nov 02 20:00:22.127568 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.127547 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc80a18b-04c3-44d6-809a-230cbde7d0db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80a18b-04c3-44d6-809a-230cbde7d0db-maas-api-service-ca\") pod \"bc80a18b-04c3-44d6-809a-230cbde7d0db\" (UID: \"bc80a18b-04c3-44d6-809a-230cbde7d0db\") " Nov 02 20:00:22.127660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.127633 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058383f3-e72a-4f34-b231-0acbde4b855d-maas-api-service-ca" pod "058383f3-e72a-4f34-b231-0acbde4b855d" (UID: "058383f3-e72a-4f34-b231-0acbde4b855d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:22.127965 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.127944 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc80a18b-04c3-44d6-809a-230cbde7d0db-maas-api-service-ca" pod "bc80a18b-04c3-44d6-809a-230cbde7d0db" (UID: "bc80a18b-04c3-44d6-809a-230cbde7d0db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:22.128193 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.128178 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7bvq\" (UniqueName: \"kubernetes.io/projected/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-kube-api-access-j7bvq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.128236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.128196 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e5bb7d-7b64-4088-9ffd-7ce1ba735894-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.128236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.128210 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wk9f\" (UniqueName: \"kubernetes.io/projected/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-kube-api-access-5wk9f\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.128236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.128219 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058383f3-e72a-4f34-b231-0acbde4b855d-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.128236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.128229 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a43c6d-a71c-4a48-a5b8-a16c10476d86-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.128402 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.128240 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80a18b-04c3-44d6-809a-230cbde7d0db-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.129433 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.129412 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc80a18b-04c3-44d6-809a-230cbde7d0db-kube-api-access-bsd5f" pod "bc80a18b-04c3-44d6-809a-230cbde7d0db" (UID: "bc80a18b-04c3-44d6-809a-230cbde7d0db"). InnerVolumeSpecName "kube-api-access-bsd5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:22.129610 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.129585 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058383f3-e72a-4f34-b231-0acbde4b855d-kube-api-access-4plr8" pod "058383f3-e72a-4f34-b231-0acbde4b855d" (UID: "058383f3-e72a-4f34-b231-0acbde4b855d"). InnerVolumeSpecName "kube-api-access-4plr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:22.185654 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.185617 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" path="/var/lib/kubelet/pods/789d89be-2cd9-481f-a5ed-2590ff0c163c/volumes" Nov 02 20:00:22.186177 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.186155 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" path="/var/lib/kubelet/pods/a78d71ed-3361-4b0e-98fc-c3cb9f174a02/volumes" Nov 02 20:00:22.186651 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.186635 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" path="/var/lib/kubelet/pods/a96626ff-f0c1-42d2-a7ba-21d659e5313a/volumes" Nov 02 20:00:22.229948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.229907 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d12a6a-61da-4204-b826-75da5bb34463-kube-api-access-n2jtn\" (UniqueName: \"kubernetes.io/projected/c0d12a6a-61da-4204-b826-75da5bb34463-kube-api-access-n2jtn\") pod \"c0d12a6a-61da-4204-b826-75da5bb34463\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " Nov 02 20:00:22.230099 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.230018 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d12a6a-61da-4204-b826-75da5bb34463-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d12a6a-61da-4204-b826-75da5bb34463-maas-api-service-ca\") pod \"c0d12a6a-61da-4204-b826-75da5bb34463\" (UID: \"c0d12a6a-61da-4204-b826-75da5bb34463\") " Nov 02 20:00:22.230476 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.230444 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d12a6a-61da-4204-b826-75da5bb34463-maas-api-service-ca" pod "c0d12a6a-61da-4204-b826-75da5bb34463" (UID: "c0d12a6a-61da-4204-b826-75da5bb34463"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:22.231088 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.231065 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d12a6a-61da-4204-b826-75da5bb34463-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.231184 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.231094 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plr8\" (UniqueName: \"kubernetes.io/projected/058383f3-e72a-4f34-b231-0acbde4b855d-kube-api-access-4plr8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.231184 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.231111 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsd5f\" (UniqueName: \"kubernetes.io/projected/bc80a18b-04c3-44d6-809a-230cbde7d0db-kube-api-access-bsd5f\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.234073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.234045 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d12a6a-61da-4204-b826-75da5bb34463-kube-api-access-n2jtn" pod "c0d12a6a-61da-4204-b826-75da5bb34463" (UID: "c0d12a6a-61da-4204-b826-75da5bb34463"). InnerVolumeSpecName "kube-api-access-n2jtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:22.335792 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.335759 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2jtn\" (UniqueName: \"kubernetes.io/projected/c0d12a6a-61da-4204-b826-75da5bb34463-kube-api-access-n2jtn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.382937 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.382895 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss"] Nov 02 20:00:22.389525 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.389490 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-23a9f5-29894160-6hkss"] Nov 02 20:00:22.519191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.519156 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf"] Nov 02 20:00:22.633345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.633307 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" Nov 02 20:00:22.633345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.633336 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4" event={"ID":"c0d12a6a-61da-4204-b826-75da5bb34463","Type":"ContainerDied","Data":"5178eca9855b2edafdc97783c996d9db58c20e8c8f35d7cd5b0e633e0d7175ec"} Nov 02 20:00:22.633871 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.633408 4966 scope.go:122] "RemoveContainer" containerID="4f88140e0789b3c66bdd43df7507106928dce41aeab67b020f2aed6eb3d04f50" Nov 02 20:00:22.637111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.636942 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" Nov 02 20:00:22.637111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.636991 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl" event={"ID":"a8a43c6d-a71c-4a48-a5b8-a16c10476d86","Type":"ContainerDied","Data":"6dd0a09f9ec88bfb8a213c8980164b2ac1c5828165461c457d8cce0c2215c7b5"} Nov 02 20:00:22.639241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.639068 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" Nov 02 20:00:22.639241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.639186 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6" event={"ID":"42e5bb7d-7b64-4088-9ffd-7ce1ba735894","Type":"ContainerDied","Data":"e9073d98bbcb4d4203f6366ceacf05f54a646fd5dadd276ca750f9e61f0a101b"} Nov 02 20:00:22.642408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.642381 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" event={"ID":"058383f3-e72a-4f34-b231-0acbde4b855d","Type":"ContainerDied","Data":"6ef731e1954371f492e5d699b4b8eaf5de064f8093bc92202207e8ae5dd55cac"} Nov 02 20:00:22.642523 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.642480 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h" Nov 02 20:00:22.647361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.647054 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" event={"ID":"bc80a18b-04c3-44d6-809a-230cbde7d0db","Type":"ContainerDied","Data":"f4ed032dfae07c552d50fea1063c1e7eaa5643a544275def815393581d92d4f8"} Nov 02 20:00:22.647361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.647136 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf" Nov 02 20:00:22.652527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.652503 4966 scope.go:122] "RemoveContainer" containerID="a49cfb5b0834fb8f8dfc25ea720ec9b2851979328c5eea52397b1b24808ddd05" Nov 02 20:00:22.677991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.677969 4966 scope.go:122] "RemoveContainer" containerID="82869440cb6a4c3560618a1fbf0d13add2988bcbc21424cccbcb9d544163cd86" Nov 02 20:00:22.690835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.690813 4966 scope.go:122] "RemoveContainer" containerID="e3eb3c62b0f3b124f727c177f6d713a3b3d84f140cf90ed491aeebc93428a125" Nov 02 20:00:22.740967 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.740937 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:22.750526 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.750500 4966 scope.go:122] "RemoveContainer" containerID="d9d46ee91623c0ffb65fcec86b0178ea197b1dc88a65c35723876aabe55d273f" Nov 02 20:00:22.837892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.837811 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8"] Nov 02 20:00:22.842088 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.842053 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e43a40-2677-4be6-9327-2e3424252de1-kube-api-access-dn2lr\" (UniqueName: \"kubernetes.io/projected/33e43a40-2677-4be6-9327-2e3424252de1-kube-api-access-dn2lr\") pod \"33e43a40-2677-4be6-9327-2e3424252de1\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " Nov 02 20:00:22.842214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.842120 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e43a40-2677-4be6-9327-2e3424252de1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e43a40-2677-4be6-9327-2e3424252de1-maas-api-service-ca\") pod \"33e43a40-2677-4be6-9327-2e3424252de1\" (UID: \"33e43a40-2677-4be6-9327-2e3424252de1\") " Nov 02 20:00:22.864457 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.853565 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e43a40-2677-4be6-9327-2e3424252de1-maas-api-service-ca" pod "33e43a40-2677-4be6-9327-2e3424252de1" (UID: "33e43a40-2677-4be6-9327-2e3424252de1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:22.873406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.871934 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e43a40-2677-4be6-9327-2e3424252de1-kube-api-access-dn2lr" pod "33e43a40-2677-4be6-9327-2e3424252de1" (UID: "33e43a40-2677-4be6-9327-2e3424252de1"). InnerVolumeSpecName "kube-api-access-dn2lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:22.967204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.967161 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn2lr\" (UniqueName: \"kubernetes.io/projected/33e43a40-2677-4be6-9327-2e3424252de1-kube-api-access-dn2lr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:22.967403 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:22.967205 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e43a40-2677-4be6-9327-2e3424252de1-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:23.035700 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.035667 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:23.146219 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.146118 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf"] Nov 02 20:00:23.151844 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.151809 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1d869e-29894160-74kwf"] Nov 02 20:00:23.169708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.169674 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b514b7f-b2f8-4dc2-82cc-285eedff1071-kube-api-access-8hldz\" (UniqueName: \"kubernetes.io/projected/5b514b7f-b2f8-4dc2-82cc-285eedff1071-kube-api-access-8hldz\") pod \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " Nov 02 20:00:23.169856 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.169780 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b514b7f-b2f8-4dc2-82cc-285eedff1071-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b514b7f-b2f8-4dc2-82cc-285eedff1071-maas-api-service-ca\") pod \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\" (UID: \"5b514b7f-b2f8-4dc2-82cc-285eedff1071\") " Nov 02 20:00:23.170147 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.170123 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b514b7f-b2f8-4dc2-82cc-285eedff1071-maas-api-service-ca" pod "5b514b7f-b2f8-4dc2-82cc-285eedff1071" (UID: "5b514b7f-b2f8-4dc2-82cc-285eedff1071"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:23.170486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.170460 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b514b7f-b2f8-4dc2-82cc-285eedff1071-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:23.171956 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.171937 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b514b7f-b2f8-4dc2-82cc-285eedff1071-kube-api-access-8hldz" pod "5b514b7f-b2f8-4dc2-82cc-285eedff1071" (UID: "5b514b7f-b2f8-4dc2-82cc-285eedff1071"). InnerVolumeSpecName "kube-api-access-8hldz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:23.249715 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.249680 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6"] Nov 02 20:00:23.254858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.254830 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-32abda-29894160-szxt6"] Nov 02 20:00:23.272444 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.272412 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hldz\" (UniqueName: \"kubernetes.io/projected/5b514b7f-b2f8-4dc2-82cc-285eedff1071-kube-api-access-8hldz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:23.357597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.357552 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl"] Nov 02 20:00:23.361931 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.361900 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8479ff-29894160-spqgl"] Nov 02 20:00:23.563705 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.563659 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h"] Nov 02 20:00:23.568111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.568076 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894160-2nv6h"] Nov 02 20:00:23.674597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.674554 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" event={"ID":"5b514b7f-b2f8-4dc2-82cc-285eedff1071","Type":"ContainerDied","Data":"ba3e84fe47934756080b786604392ba9f538933d65aaacb15dc0daee8c1cc7f6"} Nov 02 20:00:23.675137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.674631 4966 scope.go:122] "RemoveContainer" containerID="481e4b32176818cf8132e10e79c4667f7e515664487129c3aeb86ce320cd67ac" Nov 02 20:00:23.675137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.674786 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8" Nov 02 20:00:23.681410 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.681379 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4"] Nov 02 20:00:23.686584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.686561 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" Nov 02 20:00:23.686756 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.686722 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf" event={"ID":"33e43a40-2677-4be6-9327-2e3424252de1","Type":"ContainerDied","Data":"b8ca07c649aaa3b69c4b5e6f417b943454186d2e09245a1e4ffa27818c2b00fc"} Nov 02 20:00:23.688151 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.688131 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-43ceaa-29894160-bn6g4"] Nov 02 20:00:23.696717 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.696698 4966 scope.go:122] "RemoveContainer" containerID="655f7f45c110556817c9cde468d8d99f068f296cd9f71c32e95621ec9e2a9f22" Nov 02 20:00:23.887192 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.887110 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf"] Nov 02 20:00:23.891217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:23.891191 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b0cc24-29894160-p9wmf"] Nov 02 20:00:24.056327 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.053737 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8"] Nov 02 20:00:24.060321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.060227 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-89a22a-29894160-ggng8"] Nov 02 20:00:24.188355 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.188252 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" path="/var/lib/kubelet/pods/058383f3-e72a-4f34-b231-0acbde4b855d/volumes" Nov 02 20:00:24.188779 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.188761 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" path="/var/lib/kubelet/pods/149c68c2-f8a0-4d24-8e66-e2abcfc3aacf/volumes" Nov 02 20:00:24.189171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.189156 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e43a40-2677-4be6-9327-2e3424252de1" path="/var/lib/kubelet/pods/33e43a40-2677-4be6-9327-2e3424252de1/volumes" Nov 02 20:00:24.189883 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.189868 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" path="/var/lib/kubelet/pods/42e5bb7d-7b64-4088-9ffd-7ce1ba735894/volumes" Nov 02 20:00:24.190278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.190264 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" path="/var/lib/kubelet/pods/5b514b7f-b2f8-4dc2-82cc-285eedff1071/volumes" Nov 02 20:00:24.190717 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.190703 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" path="/var/lib/kubelet/pods/a8a43c6d-a71c-4a48-a5b8-a16c10476d86/volumes" Nov 02 20:00:24.191367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.191353 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" path="/var/lib/kubelet/pods/bc80a18b-04c3-44d6-809a-230cbde7d0db/volumes" Nov 02 20:00:24.191740 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:24.191727 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" path="/var/lib/kubelet/pods/c0d12a6a-61da-4204-b826-75da5bb34463/volumes" Nov 02 20:00:44.754944 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:00:44.754898 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:00:56.049061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.049024 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7"] Nov 02 20:00:56.094812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.094777 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88623-f4nw7"] Nov 02 20:00:56.185634 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.185597 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda04212-4462-46d2-b818-5afac0270838" path="/var/lib/kubelet/pods/dda04212-4462-46d2-b818-5afac0270838/volumes" Nov 02 20:00:56.555453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.555415 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw"] Nov 02 20:00:56.556135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556112 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:00:56.556135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556129 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556146 4966 cpu_manager.go:414] "containerMap: removing container" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556151 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556157 4966 cpu_manager.go:414] "containerMap: removing container" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556163 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556171 4966 cpu_manager.go:414] "containerMap: removing container" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556176 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556182 4966 cpu_manager.go:414] "containerMap: removing container" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556187 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556198 4966 cpu_manager.go:414] "containerMap: removing container" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556202 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556209 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556213 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556219 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556223 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556233 4966 cpu_manager.go:414] "containerMap: removing container" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556237 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556242 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556247 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556253 4966 cpu_manager.go:414] "containerMap: removing container" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556257 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556264 4966 cpu_manager.go:414] "containerMap: removing container" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556270 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556278 4966 cpu_manager.go:414] "containerMap: removing container" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556282 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556287 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556292 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556323 4966 cpu_manager.go:414] "containerMap: removing container" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556330 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556346 4966 cpu_manager.go:414] "containerMap: removing container" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556351 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556358 4966 cpu_manager.go:414] "containerMap: removing container" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556363 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556370 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556375 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556381 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.556368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556385 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556400 4966 cpu_manager.go:414] "containerMap: removing container" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556404 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556413 4966 cpu_manager.go:414] "containerMap: removing container" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556418 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556424 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556428 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556434 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556438 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556445 4966 cpu_manager.go:414] "containerMap: removing container" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556449 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556457 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556461 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556468 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556472 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556481 4966 cpu_manager.go:414] "containerMap: removing container" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556485 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556496 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556502 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556509 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556514 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556523 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556527 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556534 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556538 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556656 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556664 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556672 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556678 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556684 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556690 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556695 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556701 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556706 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556712 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556720 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556725 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556730 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556735 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556740 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556747 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f5dae8-66ca-497f-aec4-220ffc3ac4b8" containerName="istio-proxy" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556752 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556758 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556764 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556770 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556776 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556783 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556787 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556791 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556798 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556803 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556809 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556815 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e015108d-dcbb-4fec-a3a9-3a7fd0f8178e" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556820 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556827 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556831 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556837 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556841 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:00:56.557574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.556847 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:00:56.561809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.561793 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.612363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.612331 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw"] Nov 02 20:00:56.653179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.653134 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:00:56.653179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.653134 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:00:56.653179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.653172 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:00:56.655870 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.655846 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:00:56.703613 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.703574 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42tr\" (UniqueName: \"kubernetes.io/projected/5edbc517-0b2b-4c36-ab24-44bc97e86b38-kube-api-access-s42tr\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.703791 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.703775 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.703855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.703832 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.804763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.804727 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.804958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.804787 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.804958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.804893 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42tr\" (UniqueName: \"kubernetes.io/projected/5edbc517-0b2b-4c36-ab24-44bc97e86b38-kube-api-access-s42tr\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.805672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.805612 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.807546 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.807520 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.857370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.857341 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42tr\" (UniqueName: \"kubernetes.io/projected/5edbc517-0b2b-4c36-ab24-44bc97e86b38-kube-api-access-s42tr\") pod \"maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:56.876101 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:56.876077 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:00:57.252331 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:00:57.252270 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edbc517_0b2b_4c36_ab24_44bc97e86b38.slice/crio-fad9ca297f5e13cc63f1cf7ebaf3e83d71e59bb4f1f34408dbdcd977503bf2ba WatchSource:0}: Error finding container fad9ca297f5e13cc63f1cf7ebaf3e83d71e59bb4f1f34408dbdcd977503bf2ba: Status 404 returned error can't find the container with id fad9ca297f5e13cc63f1cf7ebaf3e83d71e59bb4f1f34408dbdcd977503bf2ba Nov 02 20:00:57.252684 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:57.252592 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw"] Nov 02 20:00:58.012545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:58.012506 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" event={"ID":"5edbc517-0b2b-4c36-ab24-44bc97e86b38","Type":"ContainerStarted","Data":"dd607ea0c4a36d5d68444efb5b5089a685c8a9c2cc87e229cd62010b5e750df9"} Nov 02 20:00:58.012722 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:58.012564 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" event={"ID":"5edbc517-0b2b-4c36-ab24-44bc97e86b38","Type":"ContainerStarted","Data":"fad9ca297f5e13cc63f1cf7ebaf3e83d71e59bb4f1f34408dbdcd977503bf2ba"} Nov 02 20:00:58.120604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:58.120548 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" podStartSLOduration=2.120533774 podStartE2EDuration="2.120533774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:58.116581204 +0000 UTC m=+1964.065237047" watchObservedRunningTime="2026-11-02 20:00:58.120533774 +0000 UTC m=+1964.069189613" Nov 02 20:00:59.023833 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:59.023792 4966 generic.go:356] "Generic (PLEG): container finished" podID="5edbc517-0b2b-4c36-ab24-44bc97e86b38" containerID="dd607ea0c4a36d5d68444efb5b5089a685c8a9c2cc87e229cd62010b5e750df9" exitCode=0 Nov 02 20:00:59.024240 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:00:59.023848 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" event={"ID":"5edbc517-0b2b-4c36-ab24-44bc97e86b38","Type":"ContainerDied","Data":"dd607ea0c4a36d5d68444efb5b5089a685c8a9c2cc87e229cd62010b5e750df9"} Nov 02 20:01:00.252632 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.252607 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:01:00.357998 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.357904 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-openshift-ca\") pod \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " Nov 02 20:01:00.358165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.358011 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-xks-ca\") pod \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " Nov 02 20:01:00.358165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.358125 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edbc517-0b2b-4c36-ab24-44bc97e86b38-kube-api-access-s42tr\" (UniqueName: \"kubernetes.io/projected/5edbc517-0b2b-4c36-ab24-44bc97e86b38-kube-api-access-s42tr\") pod \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\" (UID: \"5edbc517-0b2b-4c36-ab24-44bc97e86b38\") " Nov 02 20:01:00.358576 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.358540 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-openshift-ca" pod "5edbc517-0b2b-4c36-ab24-44bc97e86b38" (UID: "5edbc517-0b2b-4c36-ab24-44bc97e86b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:00.358817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.358794 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:00.361519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.361484 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-xks-ca" pod "5edbc517-0b2b-4c36-ab24-44bc97e86b38" (UID: "5edbc517-0b2b-4c36-ab24-44bc97e86b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:01:00.361663 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.361515 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edbc517-0b2b-4c36-ab24-44bc97e86b38-kube-api-access-s42tr" pod "5edbc517-0b2b-4c36-ab24-44bc97e86b38" (UID: "5edbc517-0b2b-4c36-ab24-44bc97e86b38"). InnerVolumeSpecName "kube-api-access-s42tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:00.460395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.460361 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edbc517-0b2b-4c36-ab24-44bc97e86b38-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:00.460395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:00.460393 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42tr\" (UniqueName: \"kubernetes.io/projected/5edbc517-0b2b-4c36-ab24-44bc97e86b38-kube-api-access-s42tr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:01.044694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:01.044660 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" event={"ID":"5edbc517-0b2b-4c36-ab24-44bc97e86b38","Type":"ContainerDied","Data":"fad9ca297f5e13cc63f1cf7ebaf3e83d71e59bb4f1f34408dbdcd977503bf2ba"} Nov 02 20:01:01.044694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:01.044687 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw" Nov 02 20:01:01.044912 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:01.044708 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad9ca297f5e13cc63f1cf7ebaf3e83d71e59bb4f1f34408dbdcd977503bf2ba" Nov 02 20:01:14.754877 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:01:14.754826 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:01:18.086321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:18.086278 4966 scope.go:122] "RemoveContainer" containerID="b3c5b6d64727eb8d339a8918aa4154aff97210d94610cf388ef92ee5b96da09d" Nov 02 20:01:20.420880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.420842 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk"] Nov 02 20:01:20.421623 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421606 4966 cpu_manager.go:414] "containerMap: removing container" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:01:20.421679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421625 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc80a18b-04c3-44d6-809a-230cbde7d0db" containerName="cleanup" Nov 02 20:01:20.421679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421636 4966 cpu_manager.go:414] "containerMap: removing container" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:01:20.421679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421641 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149c68c2-f8a0-4d24-8e66-e2abcfc3aacf" containerName="cleanup" Nov 02 20:01:20.421679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421663 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:01:20.421679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421670 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b514b7f-b2f8-4dc2-82cc-285eedff1071" containerName="cleanup" Nov 02 20:01:20.421679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421678 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421683 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d12a6a-61da-4204-b826-75da5bb34463" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421693 4966 cpu_manager.go:414] "containerMap: removing container" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421698 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058383f3-e72a-4f34-b231-0acbde4b855d" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421714 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5edbc517-0b2b-4c36-ab24-44bc97e86b38" containerName="revoke-keys" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421719 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edbc517-0b2b-4c36-ab24-44bc97e86b38" containerName="revoke-keys" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421727 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421731 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421738 4966 cpu_manager.go:414] "containerMap: removing container" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421742 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421747 4966 cpu_manager.go:414] "containerMap: removing container" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421751 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e43a40-2677-4be6-9327-2e3424252de1" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421768 4966 cpu_manager.go:414] "containerMap: removing container" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421772 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421900 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="789d89be-2cd9-481f-a5ed-2590ff0c163c" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421911 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a43c6d-a71c-4a48-a5b8-a16c10476d86" containerName="cleanup" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421919 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edbc517-0b2b-4c36-ab24-44bc97e86b38" containerName="revoke-keys" Nov 02 20:01:20.421923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421926 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78d71ed-3361-4b0e-98fc-c3cb9f174a02" containerName="cleanup" Nov 02 20:01:20.422453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421932 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e5bb7d-7b64-4088-9ffd-7ce1ba735894" containerName="cleanup" Nov 02 20:01:20.422453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421940 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e09a4f-2cc9-4044-8ddd-18eedfd04ff4" containerName="cleanup" Nov 02 20:01:20.422453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.421949 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96626ff-f0c1-42d2-a7ba-21d659e5313a" containerName="cleanup" Nov 02 20:01:20.428143 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.428124 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.465373 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.465338 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.465565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.465485 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.465708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.465689 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gln\" (UniqueName: \"kubernetes.io/projected/ca75052c-7bda-4920-8b53-4cd06691abc9-kube-api-access-v6gln\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.476612 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.476579 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:01:20.477037 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.476638 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:01:20.477923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.477898 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:01:20.478217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.477918 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:01:20.484601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.484569 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk"] Nov 02 20:01:20.567064 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.567020 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gln\" (UniqueName: \"kubernetes.io/projected/ca75052c-7bda-4920-8b53-4cd06691abc9-kube-api-access-v6gln\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.567275 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.567084 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.567275 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.567184 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.568033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.568006 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.569668 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.569641 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.619711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.619681 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gln\" (UniqueName: \"kubernetes.io/projected/ca75052c-7bda-4920-8b53-4cd06691abc9-kube-api-access-v6gln\") pod \"maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:20.743176 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:20.743140 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:21.119769 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:21.119727 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk"] Nov 02 20:01:21.119903 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:01:21.119832 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca75052c_7bda_4920_8b53_4cd06691abc9.slice/crio-5f2f50bbb985add52de0556e711894ea938c76f810c8efd2be169587471f3c0b WatchSource:0}: Error finding container 5f2f50bbb985add52de0556e711894ea938c76f810c8efd2be169587471f3c0b: Status 404 returned error can't find the container with id 5f2f50bbb985add52de0556e711894ea938c76f810c8efd2be169587471f3c0b Nov 02 20:01:21.259566 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:21.259533 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" event={"ID":"ca75052c-7bda-4920-8b53-4cd06691abc9","Type":"ContainerStarted","Data":"5f2f50bbb985add52de0556e711894ea938c76f810c8efd2be169587471f3c0b"} Nov 02 20:01:22.269332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:22.269269 4966 generic.go:356] "Generic (PLEG): container finished" podID="ca75052c-7bda-4920-8b53-4cd06691abc9" containerID="a3bf37710cdce76ed30d790e10da166b2b1b17bb5a26370b14cdd7f3ec1a4b06" exitCode=0 Nov 02 20:01:22.269764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:22.269364 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" event={"ID":"ca75052c-7bda-4920-8b53-4cd06691abc9","Type":"ContainerDied","Data":"a3bf37710cdce76ed30d790e10da166b2b1b17bb5a26370b14cdd7f3ec1a4b06"} Nov 02 20:01:23.454693 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.454665 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:23.507398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.507365 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca75052c-7bda-4920-8b53-4cd06691abc9-kube-api-access-v6gln\" (UniqueName: \"kubernetes.io/projected/ca75052c-7bda-4920-8b53-4cd06691abc9-kube-api-access-v6gln\") pod \"ca75052c-7bda-4920-8b53-4cd06691abc9\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " Nov 02 20:01:23.507606 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.507410 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-xks-ca\") pod \"ca75052c-7bda-4920-8b53-4cd06691abc9\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " Nov 02 20:01:23.507606 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.507513 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-openshift-ca\") pod \"ca75052c-7bda-4920-8b53-4cd06691abc9\" (UID: \"ca75052c-7bda-4920-8b53-4cd06691abc9\") " Nov 02 20:01:23.507895 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.507868 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-openshift-ca" pod "ca75052c-7bda-4920-8b53-4cd06691abc9" (UID: "ca75052c-7bda-4920-8b53-4cd06691abc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:23.508088 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.508065 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:23.509769 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.509736 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-xks-ca" pod "ca75052c-7bda-4920-8b53-4cd06691abc9" (UID: "ca75052c-7bda-4920-8b53-4cd06691abc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:01:23.509858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.509835 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca75052c-7bda-4920-8b53-4cd06691abc9-kube-api-access-v6gln" pod "ca75052c-7bda-4920-8b53-4cd06691abc9" (UID: "ca75052c-7bda-4920-8b53-4cd06691abc9"). InnerVolumeSpecName "kube-api-access-v6gln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:23.609975 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.609884 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gln\" (UniqueName: \"kubernetes.io/projected/ca75052c-7bda-4920-8b53-4cd06691abc9-kube-api-access-v6gln\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:23.609975 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:23.609914 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca75052c-7bda-4920-8b53-4cd06691abc9-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:24.286809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:24.286774 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" Nov 02 20:01:24.286809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:24.286806 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk" event={"ID":"ca75052c-7bda-4920-8b53-4cd06691abc9","Type":"ContainerDied","Data":"5f2f50bbb985add52de0556e711894ea938c76f810c8efd2be169587471f3c0b"} Nov 02 20:01:24.287028 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:24.286856 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2f50bbb985add52de0556e711894ea938c76f810c8efd2be169587471f3c0b" Nov 02 20:01:30.047264 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:30.047224 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq"] Nov 02 20:01:30.105394 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:30.103471 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-830941e5-ftvvq"] Nov 02 20:01:30.198133 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:30.198099 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c" path="/var/lib/kubelet/pods/86e9f0f3-d6a0-421f-9384-d7f0d89e1b3c/volumes" Nov 02 20:01:39.017203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:39.017164 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh"] Nov 02 20:01:39.017764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:39.017460 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" containerID="cri-o://5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25" gracePeriod=30 Nov 02 20:01:39.067694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:39.067655 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4"] Nov 02 20:01:39.067993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:39.067912 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" containerID="cri-o://7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8" gracePeriod=30 Nov 02 20:01:41.793676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:41.793640 4966 patch_prober.go:29] interesting pod/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:01:41.794084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:41.793718 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:01:42.008336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.008290 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26"] Nov 02 20:01:42.009014 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.008999 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ca75052c-7bda-4920-8b53-4cd06691abc9" containerName="revoke-keys" Nov 02 20:01:42.009014 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.009015 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca75052c-7bda-4920-8b53-4cd06691abc9" containerName="revoke-keys" Nov 02 20:01:42.009171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.009159 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca75052c-7bda-4920-8b53-4cd06691abc9" containerName="revoke-keys" Nov 02 20:01:42.012967 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.012950 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.066824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.066740 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26"] Nov 02 20:01:42.106801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.106758 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:01:42.106999 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.106819 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:01:42.108015 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.107977 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:01:42.108015 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.108000 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:01:42.115624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.115586 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.115761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.115646 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.115885 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.115862 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnd7s\" (UniqueName: \"kubernetes.io/projected/1a9f8715-6b8f-4350-aa36-8859e468a1c1-kube-api-access-xnd7s\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.217240 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.217208 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd7s\" (UniqueName: \"kubernetes.io/projected/1a9f8715-6b8f-4350-aa36-8859e468a1c1-kube-api-access-xnd7s\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.217464 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.217434 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.217887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.217860 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.220088 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.220056 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.221494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.221469 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.272280 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.272248 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnd7s\" (UniqueName: \"kubernetes.io/projected/1a9f8715-6b8f-4350-aa36-8859e468a1c1-kube-api-access-xnd7s\") pod \"maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.328814 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.328724 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:42.710767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.710731 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26"] Nov 02 20:01:42.711561 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:01:42.711533 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9f8715_6b8f_4350_aa36_8859e468a1c1.slice/crio-fb0560ac55b8b0ed5aadff926641c4b04c42c7cf67ade107585465559fd32275 WatchSource:0}: Error finding container fb0560ac55b8b0ed5aadff926641c4b04c42c7cf67ade107585465559fd32275: Status 404 returned error can't find the container with id fb0560ac55b8b0ed5aadff926641c4b04c42c7cf67ade107585465559fd32275 Nov 02 20:01:42.716887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.716857 4966 patch_prober.go:29] interesting pod/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:01:42.716991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:42.716911 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:01:43.459552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:43.459518 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" event={"ID":"1a9f8715-6b8f-4350-aa36-8859e468a1c1","Type":"ContainerStarted","Data":"32a049a8d0341cbef522a59939c6bb303fdbbd3c049948ff0d6230b0a6c2a85a"} Nov 02 20:01:43.459961 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:43.459569 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" event={"ID":"1a9f8715-6b8f-4350-aa36-8859e468a1c1","Type":"ContainerStarted","Data":"fb0560ac55b8b0ed5aadff926641c4b04c42c7cf67ade107585465559fd32275"} Nov 02 20:01:43.569239 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:43.569169 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" podStartSLOduration=2.569153022 podStartE2EDuration="2.569153022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:43.564694568 +0000 UTC m=+2009.513350408" watchObservedRunningTime="2026-11-02 20:01:43.569153022 +0000 UTC m=+2009.517808858" Nov 02 20:01:44.366393 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.366361 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 20:01:44.411862 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.411835 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 20:01:44.457529 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457500 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-data\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.457709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457628 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-credential-socket\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.457709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457655 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-istio-token\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-istio-token\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.457857 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457773 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc150de8-bfd9-468d-b366-91177eadded8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc150de8-bfd9-468d-b366-91177eadded8-istio-podinfo\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.457857 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457805 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-data" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.457857 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457815 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-kube-api-access-s4k8v\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-kube-api-access-s4k8v\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.458023 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457902 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-credential-socket" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.458023 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.457935 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc150de8-bfd9-468d-b366-91177eadded8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc150de8-bfd9-468d-b366-91177eadded8-istiod-ca-cert\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.458127 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458019 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-socket\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.458127 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458068 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-certs\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.458127 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458119 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-envoy\") pod \"cc150de8-bfd9-468d-b366-91177eadded8\" (UID: \"cc150de8-bfd9-468d-b366-91177eadded8\") " Nov 02 20:01:44.458426 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458337 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-socket" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.458426 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458357 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc150de8-bfd9-468d-b366-91177eadded8-istiod-ca-cert" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:44.458643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458467 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-certs" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.458916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458890 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.459022 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458923 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc150de8-bfd9-468d-b366-91177eadded8-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.459022 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458939 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.459022 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458956 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.459022 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.458971 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.460336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.460310 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-istio-token" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:44.460680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.460348 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-kube-api-access-s4k8v" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "kube-api-access-s4k8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:44.460680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.460375 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc150de8-bfd9-468d-b366-91177eadded8-istio-podinfo" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:01:44.463691 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.463665 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-envoy" pod "cc150de8-bfd9-468d-b366-91177eadded8" (UID: "cc150de8-bfd9-468d-b366-91177eadded8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.470388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.470361 4966 generic.go:356] "Generic (PLEG): container finished" podID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerID="7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8" exitCode=0 Nov 02 20:01:44.470494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.470402 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" Nov 02 20:01:44.470494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.470437 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" event={"ID":"31e54d54-22dc-48f5-91b2-dc71a057d0e9","Type":"ContainerDied","Data":"7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8"} Nov 02 20:01:44.470494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.470491 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4" event={"ID":"31e54d54-22dc-48f5-91b2-dc71a057d0e9","Type":"ContainerDied","Data":"854b6030ec34a14e78d61bba2dadd9a88d079efd62f2cdd3e8a7f5b141c18127"} Nov 02 20:01:44.470614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.470513 4966 scope.go:122] "RemoveContainer" containerID="7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8" Nov 02 20:01:44.472365 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.472342 4966 generic.go:356] "Generic (PLEG): container finished" podID="1a9f8715-6b8f-4350-aa36-8859e468a1c1" containerID="32a049a8d0341cbef522a59939c6bb303fdbbd3c049948ff0d6230b0a6c2a85a" exitCode=0 Nov 02 20:01:44.472480 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.472425 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" event={"ID":"1a9f8715-6b8f-4350-aa36-8859e468a1c1","Type":"ContainerDied","Data":"32a049a8d0341cbef522a59939c6bb303fdbbd3c049948ff0d6230b0a6c2a85a"} Nov 02 20:01:44.474108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.474084 4966 generic.go:356] "Generic (PLEG): container finished" podID="cc150de8-bfd9-468d-b366-91177eadded8" containerID="5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25" exitCode=0 Nov 02 20:01:44.476428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.474227 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" Nov 02 20:01:44.476428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.474243 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" event={"ID":"cc150de8-bfd9-468d-b366-91177eadded8","Type":"ContainerDied","Data":"5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25"} Nov 02 20:01:44.476428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.474278 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh" event={"ID":"cc150de8-bfd9-468d-b366-91177eadded8","Type":"ContainerDied","Data":"a7e0a1a3c5316e68f15a6c21ad8822d30481a8130e683088b0d6925b2179e86c"} Nov 02 20:01:44.489233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.489211 4966 scope.go:122] "RemoveContainer" containerID="7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8" Nov 02 20:01:44.489634 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:01:44.489601 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8\": container with ID starting with 7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8 not found: ID does not exist" containerID="7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8" Nov 02 20:01:44.489719 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.489644 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8"} err="failed to get container status \"7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8\": rpc error: code = NotFound desc = could not find container \"7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8\": container with ID starting with 7f2d17251e56184adb4271c52728ebb035195f37c1c01f180cb866a0cce87aa8 not found: ID does not exist" Nov 02 20:01:44.489719 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.489668 4966 scope.go:122] "RemoveContainer" containerID="5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25" Nov 02 20:01:44.501553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.501529 4966 scope.go:122] "RemoveContainer" containerID="5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25" Nov 02 20:01:44.501865 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:01:44.501846 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25\": container with ID starting with 5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25 not found: ID does not exist" containerID="5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25" Nov 02 20:01:44.501943 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.501872 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25"} err="failed to get container status \"5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25\": rpc error: code = NotFound desc = could not find container \"5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25\": container with ID starting with 5ca6e0a7682f9c1c7b80947d9e42f35fe3108a713ac37fcb17dad02f00809e25 not found: ID does not exist" Nov 02 20:01:44.559802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.559752 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-credential-socket\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.559968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.559869 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istiod-ca-cert\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.559968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.559911 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-certs\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.560111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.560024 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-data\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.560175 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.560108 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-podinfo\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.560230 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.560172 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-socket\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.560230 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.560211 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-token\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-token\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.560396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.560245 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-envoy\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.560396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.560318 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-kube-api-access-jvrm4\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-kube-api-access-jvrm4\") pod \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\" (UID: \"31e54d54-22dc-48f5-91b2-dc71a057d0e9\") " Nov 02 20:01:44.560993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.560806 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-data" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.561102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561015 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc150de8-bfd9-468d-b366-91177eadded8-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.561102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561040 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4k8v\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-kube-api-access-s4k8v\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.561102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561055 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.561102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561069 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc150de8-bfd9-468d-b366-91177eadded8-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.561102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561083 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc150de8-bfd9-468d-b366-91177eadded8-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.561417 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561205 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istiod-ca-cert" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:44.561417 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561405 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-socket" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.561552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561526 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-credential-socket" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.561682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.561586 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-certs" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.563516 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.563491 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-token" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:44.563706 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.563677 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-kube-api-access-jvrm4" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "kube-api-access-jvrm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:44.564644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.564625 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-podinfo" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:01:44.564770 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.564746 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-envoy" pod "31e54d54-22dc-48f5-91b2-dc71a057d0e9" (UID: "31e54d54-22dc-48f5-91b2-dc71a057d0e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662166 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrm4\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-kube-api-access-jvrm4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662198 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662209 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662218 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662227 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662235 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662243 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.662245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.662250 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e54d54-22dc-48f5-91b2-dc71a057d0e9-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:44.732073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.732034 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh"] Nov 02 20:01:44.736194 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.736166 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5c46af-openshift-default-c86f9fbdf-8q8bh"] Nov 02 20:01:44.755962 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:01:44.755929 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:01:44.946128 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.946042 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4"] Nov 02 20:01:44.955941 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:44.955898 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-5jjn4"] Nov 02 20:01:45.690756 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.690731 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:45.777446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.777350 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9f8715-6b8f-4350-aa36-8859e468a1c1-kube-api-access-xnd7s\" (UniqueName: \"kubernetes.io/projected/1a9f8715-6b8f-4350-aa36-8859e468a1c1-kube-api-access-xnd7s\") pod \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " Nov 02 20:01:45.777446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.777447 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-xks-ca\") pod \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " Nov 02 20:01:45.777908 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.777880 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-openshift-ca\") pod \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\" (UID: \"1a9f8715-6b8f-4350-aa36-8859e468a1c1\") " Nov 02 20:01:45.779648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.778266 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-openshift-ca" pod "1a9f8715-6b8f-4350-aa36-8859e468a1c1" (UID: "1a9f8715-6b8f-4350-aa36-8859e468a1c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:45.779758 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.779650 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:45.780684 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.780664 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-xks-ca" pod "1a9f8715-6b8f-4350-aa36-8859e468a1c1" (UID: "1a9f8715-6b8f-4350-aa36-8859e468a1c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:01:45.780950 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.780913 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9f8715-6b8f-4350-aa36-8859e468a1c1-kube-api-access-xnd7s" pod "1a9f8715-6b8f-4350-aa36-8859e468a1c1" (UID: "1a9f8715-6b8f-4350-aa36-8859e468a1c1"). InnerVolumeSpecName "kube-api-access-xnd7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:45.880721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.880682 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnd7s\" (UniqueName: \"kubernetes.io/projected/1a9f8715-6b8f-4350-aa36-8859e468a1c1-kube-api-access-xnd7s\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:45.880721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:45.880718 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9f8715-6b8f-4350-aa36-8859e468a1c1-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:46.186502 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:46.186467 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" path="/var/lib/kubelet/pods/31e54d54-22dc-48f5-91b2-dc71a057d0e9/volumes" Nov 02 20:01:46.187058 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:46.186995 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc150de8-bfd9-468d-b366-91177eadded8" path="/var/lib/kubelet/pods/cc150de8-bfd9-468d-b366-91177eadded8/volumes" Nov 02 20:01:46.496587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:46.496554 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" Nov 02 20:01:46.496780 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:46.496587 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26" event={"ID":"1a9f8715-6b8f-4350-aa36-8859e468a1c1","Type":"ContainerDied","Data":"fb0560ac55b8b0ed5aadff926641c4b04c42c7cf67ade107585465559fd32275"} Nov 02 20:01:46.496780 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:46.496645 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0560ac55b8b0ed5aadff926641c4b04c42c7cf67ade107585465559fd32275" Nov 02 20:01:49.809258 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.809204 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh"] Nov 02 20:01:49.810191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810169 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1a9f8715-6b8f-4350-aa36-8859e468a1c1" containerName="revoke-keys" Nov 02 20:01:49.810191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810191 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9f8715-6b8f-4350-aa36-8859e468a1c1" containerName="revoke-keys" Nov 02 20:01:49.810359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810211 4966 cpu_manager.go:414] "containerMap: removing container" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" Nov 02 20:01:49.810359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810216 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" Nov 02 20:01:49.810359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810236 4966 cpu_manager.go:414] "containerMap: removing container" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" Nov 02 20:01:49.810359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810241 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" Nov 02 20:01:49.810579 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810376 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9f8715-6b8f-4350-aa36-8859e468a1c1" containerName="revoke-keys" Nov 02 20:01:49.810579 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810387 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e54d54-22dc-48f5-91b2-dc71a057d0e9" containerName="istio-proxy" Nov 02 20:01:49.810579 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.810395 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc150de8-bfd9-468d-b366-91177eadded8" containerName="istio-proxy" Nov 02 20:01:49.814968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.814941 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:49.867341 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.867282 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh"] Nov 02 20:01:49.907399 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.907362 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:01:49.907399 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.907378 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:01:49.909248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.909224 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:01:49.909248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.909233 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:01:49.942223 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.942182 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:49.942464 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.942398 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph56f\" (UniqueName: \"kubernetes.io/projected/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-kube-api-access-ph56f\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:49.942557 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:49.942538 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.043903 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.043866 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph56f\" (UniqueName: \"kubernetes.io/projected/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-kube-api-access-ph56f\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.044110 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.044004 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.044183 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.044140 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.045083 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.045054 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.047107 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.047080 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.097657 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.097578 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph56f\" (UniqueName: \"kubernetes.io/projected/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-kube-api-access-ph56f\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.133607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.133574 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:50.523123 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:01:50.523085 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd888ca0_3e2b_4b3f_8a98_9abaeba31d3b.slice/crio-825d6f5197572efb76f4f2a827619b0164f1fab470fb999a9f10ae608eba6f75 WatchSource:0}: Error finding container 825d6f5197572efb76f4f2a827619b0164f1fab470fb999a9f10ae608eba6f75: Status 404 returned error can't find the container with id 825d6f5197572efb76f4f2a827619b0164f1fab470fb999a9f10ae608eba6f75 Nov 02 20:01:50.524493 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.524463 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh"] Nov 02 20:01:50.543001 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:50.542964 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" event={"ID":"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b","Type":"ContainerStarted","Data":"825d6f5197572efb76f4f2a827619b0164f1fab470fb999a9f10ae608eba6f75"} Nov 02 20:01:51.553387 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:51.553350 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" event={"ID":"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b","Type":"ContainerStarted","Data":"ddecff5a8f50cac2d1347155732b50e108bb00d0176c6ead83cfc63d4a5cb7e6"} Nov 02 20:01:52.562639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.562603 4966 generic.go:356] "Generic (PLEG): container finished" podID="fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" containerID="ddecff5a8f50cac2d1347155732b50e108bb00d0176c6ead83cfc63d4a5cb7e6" exitCode=0 Nov 02 20:01:52.563050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.562687 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" event={"ID":"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b","Type":"ContainerDied","Data":"ddecff5a8f50cac2d1347155732b50e108bb00d0176c6ead83cfc63d4a5cb7e6"} Nov 02 20:01:52.756964 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.756933 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:52.887031 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.886842 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-xks-ca\") pod \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " Nov 02 20:01:52.887031 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.886941 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-openshift-ca\") pod \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " Nov 02 20:01:52.887031 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.886996 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-kube-api-access-ph56f\" (UniqueName: \"kubernetes.io/projected/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-kube-api-access-ph56f\") pod \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\" (UID: \"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b\") " Nov 02 20:01:52.887410 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.887380 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-openshift-ca" pod "fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" (UID: "fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:01:52.887776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.887730 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:52.889287 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.889264 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-kube-api-access-ph56f" pod "fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" (UID: "fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b"). InnerVolumeSpecName "kube-api-access-ph56f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:52.889548 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.889527 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-xks-ca" pod "fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" (UID: "fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:01:52.988785 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.988749 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:52.988785 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:52.988780 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph56f\" (UniqueName: \"kubernetes.io/projected/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b-kube-api-access-ph56f\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:53.573953 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:53.573916 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" Nov 02 20:01:53.574396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:53.573929 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh" event={"ID":"fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b","Type":"ContainerDied","Data":"825d6f5197572efb76f4f2a827619b0164f1fab470fb999a9f10ae608eba6f75"} Nov 02 20:01:53.574396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:01:53.574046 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="825d6f5197572efb76f4f2a827619b0164f1fab470fb999a9f10ae608eba6f75" Nov 02 20:02:01.355517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:01.355224 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6"] Nov 02 20:02:01.475959 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:01.475925 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78"] Nov 02 20:02:06.356233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.356131 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" podUID="7144b232-f9ed-4b70-979e-70e066b021ad" containerName="payload-processing" containerID="cri-o://1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542" gracePeriod=25 Nov 02 20:02:06.477167 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.477112 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" podUID="722d1f95-7b32-43ea-9007-0230a25eaca4" containerName="payload-pre-processing" containerID="cri-o://940269dda5b94f911beeade3369d5927026de291f912da591da4470b946e28d6" gracePeriod=25 Nov 02 20:02:06.640759 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.640386 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 20:02:06.710977 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.710881 4966 generic.go:356] "Generic (PLEG): container finished" podID="7144b232-f9ed-4b70-979e-70e066b021ad" containerID="1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542" exitCode=0 Nov 02 20:02:06.710977 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.710931 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" event={"ID":"7144b232-f9ed-4b70-979e-70e066b021ad","Type":"ContainerDied","Data":"1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542"} Nov 02 20:02:06.711249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.711016 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" event={"ID":"7144b232-f9ed-4b70-979e-70e066b021ad","Type":"ContainerDied","Data":"b3d6f797e9b9ff18d95a0e0c101d212a04c5694664355dd546815a9d2a0a56ea"} Nov 02 20:02:06.711249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.711049 4966 scope.go:122] "RemoveContainer" containerID="1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542" Nov 02 20:02:06.711249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.710959 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6" Nov 02 20:02:06.714993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.714961 4966 generic.go:356] "Generic (PLEG): container finished" podID="722d1f95-7b32-43ea-9007-0230a25eaca4" containerID="940269dda5b94f911beeade3369d5927026de291f912da591da4470b946e28d6" exitCode=0 Nov 02 20:02:06.715240 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.715219 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" event={"ID":"722d1f95-7b32-43ea-9007-0230a25eaca4","Type":"ContainerDied","Data":"940269dda5b94f911beeade3369d5927026de291f912da591da4470b946e28d6"} Nov 02 20:02:06.717773 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.717750 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7144b232-f9ed-4b70-979e-70e066b021ad-kube-api-access-lcgpf\" (UniqueName: \"kubernetes.io/projected/7144b232-f9ed-4b70-979e-70e066b021ad-kube-api-access-lcgpf\") pod \"7144b232-f9ed-4b70-979e-70e066b021ad\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " Nov 02 20:02:06.718262 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.717978 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7144b232-f9ed-4b70-979e-70e066b021ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7144b232-f9ed-4b70-979e-70e066b021ad-plugins-config-volume\") pod \"7144b232-f9ed-4b70-979e-70e066b021ad\" (UID: \"7144b232-f9ed-4b70-979e-70e066b021ad\") " Nov 02 20:02:06.718779 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.718667 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7144b232-f9ed-4b70-979e-70e066b021ad-plugins-config-volume" pod "7144b232-f9ed-4b70-979e-70e066b021ad" (UID: "7144b232-f9ed-4b70-979e-70e066b021ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:02:06.720999 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.720977 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7144b232-f9ed-4b70-979e-70e066b021ad-kube-api-access-lcgpf" pod "7144b232-f9ed-4b70-979e-70e066b021ad" (UID: "7144b232-f9ed-4b70-979e-70e066b021ad"). InnerVolumeSpecName "kube-api-access-lcgpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:06.749632 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.748894 4966 scope.go:122] "RemoveContainer" containerID="1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542" Nov 02 20:02:06.749632 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:02:06.749378 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542\": container with ID starting with 1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542 not found: ID does not exist" containerID="1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542" Nov 02 20:02:06.749632 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.749416 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542"} err="failed to get container status \"1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542\": rpc error: code = NotFound desc = could not find container \"1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542\": container with ID starting with 1250a93220701bfce10564ccb7432209e7a701aedea3ceaf964ad62c99a3a542 not found: ID does not exist" Nov 02 20:02:06.764360 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.764326 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 20:02:06.820084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.820055 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcgpf\" (UniqueName: \"kubernetes.io/projected/7144b232-f9ed-4b70-979e-70e066b021ad-kube-api-access-lcgpf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:06.820084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.820082 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7144b232-f9ed-4b70-979e-70e066b021ad-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:06.920829 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.920750 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722d1f95-7b32-43ea-9007-0230a25eaca4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722d1f95-7b32-43ea-9007-0230a25eaca4-plugins-config-volume\") pod \"722d1f95-7b32-43ea-9007-0230a25eaca4\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " Nov 02 20:02:06.921010 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.920986 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722d1f95-7b32-43ea-9007-0230a25eaca4-kube-api-access-bdh5m\" (UniqueName: \"kubernetes.io/projected/722d1f95-7b32-43ea-9007-0230a25eaca4-kube-api-access-bdh5m\") pod \"722d1f95-7b32-43ea-9007-0230a25eaca4\" (UID: \"722d1f95-7b32-43ea-9007-0230a25eaca4\") " Nov 02 20:02:06.921256 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.921223 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722d1f95-7b32-43ea-9007-0230a25eaca4-plugins-config-volume" pod "722d1f95-7b32-43ea-9007-0230a25eaca4" (UID: "722d1f95-7b32-43ea-9007-0230a25eaca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:02:06.921731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.921705 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/722d1f95-7b32-43ea-9007-0230a25eaca4-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:06.923466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:06.923437 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722d1f95-7b32-43ea-9007-0230a25eaca4-kube-api-access-bdh5m" pod "722d1f95-7b32-43ea-9007-0230a25eaca4" (UID: "722d1f95-7b32-43ea-9007-0230a25eaca4"). InnerVolumeSpecName "kube-api-access-bdh5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:07.022749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.022721 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdh5m\" (UniqueName: \"kubernetes.io/projected/722d1f95-7b32-43ea-9007-0230a25eaca4-kube-api-access-bdh5m\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:07.126400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.126368 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6"] Nov 02 20:02:07.130982 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.130951 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-89a22a-5d5855dfd5-2tgw6"] Nov 02 20:02:07.725880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.725852 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" Nov 02 20:02:07.726376 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.725890 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78" event={"ID":"722d1f95-7b32-43ea-9007-0230a25eaca4","Type":"ContainerDied","Data":"5a4810cf11fd09c4512e58354e763f6a5e6556ea2e5e5804bc27cfae080c0365"} Nov 02 20:02:07.726376 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.725948 4966 scope.go:122] "RemoveContainer" containerID="940269dda5b94f911beeade3369d5927026de291f912da591da4470b946e28d6" Nov 02 20:02:07.880582 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.880547 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78"] Nov 02 20:02:07.926175 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:07.926136 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89a22a-586d56595-dfq78"] Nov 02 20:02:08.185289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:08.185193 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7144b232-f9ed-4b70-979e-70e066b021ad" path="/var/lib/kubelet/pods/7144b232-f9ed-4b70-979e-70e066b021ad/volumes" Nov 02 20:02:08.185705 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:08.185682 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722d1f95-7b32-43ea-9007-0230a25eaca4" path="/var/lib/kubelet/pods/722d1f95-7b32-43ea-9007-0230a25eaca4/volumes" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.164521 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt"] Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165472 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7144b232-f9ed-4b70-979e-70e066b021ad" containerName="payload-processing" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165495 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7144b232-f9ed-4b70-979e-70e066b021ad" containerName="payload-processing" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165532 4966 cpu_manager.go:414] "containerMap: removing container" podUID="fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" containerName="revoke-keys" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165546 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" containerName="revoke-keys" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165588 4966 cpu_manager.go:414] "containerMap: removing container" podUID="722d1f95-7b32-43ea-9007-0230a25eaca4" containerName="payload-pre-processing" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165601 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722d1f95-7b32-43ea-9007-0230a25eaca4" containerName="payload-pre-processing" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165857 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7144b232-f9ed-4b70-979e-70e066b021ad" containerName="payload-processing" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165874 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" containerName="revoke-keys" Nov 02 20:02:10.168323 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.165890 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="722d1f95-7b32-43ea-9007-0230a25eaca4" containerName="payload-pre-processing" Nov 02 20:02:10.211921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.211887 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt"] Nov 02 20:02:10.212422 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.212400 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.265336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.264293 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:02:10.267907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.267831 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:02:10.267907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.267866 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:02:10.267907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.267901 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:02:10.389720 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.389392 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.389720 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.389529 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fj2q\" (UniqueName: \"kubernetes.io/projected/27652d1d-d665-4e95-b9b5-b231cb1a5202-kube-api-access-8fj2q\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.389720 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.389564 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.491351 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.491277 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.491517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.491402 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fj2q\" (UniqueName: \"kubernetes.io/projected/27652d1d-d665-4e95-b9b5-b231cb1a5202-kube-api-access-8fj2q\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.491517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.491426 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.491955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.491931 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.493877 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.493854 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.544873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.544835 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fj2q\" (UniqueName: \"kubernetes.io/projected/27652d1d-d665-4e95-b9b5-b231cb1a5202-kube-api-access-8fj2q\") pod \"maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:10.558201 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:10.558173 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:11.011613 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:11.010982 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt"] Nov 02 20:02:11.763206 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:11.763171 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" event={"ID":"27652d1d-d665-4e95-b9b5-b231cb1a5202","Type":"ContainerStarted","Data":"0e0684ab7e042dbc7ce1d43cc6d29bb32729b4d11eb870785a64f4430af011b0"} Nov 02 20:02:11.763667 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:11.763231 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" event={"ID":"27652d1d-d665-4e95-b9b5-b231cb1a5202","Type":"ContainerStarted","Data":"23966880dbdd75cdc68e307906773d531a20dcfb1c4ffa5f23ce78631c78c20a"} Nov 02 20:02:11.869727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:11.869666 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" podStartSLOduration=1.8696518370000001 podStartE2EDuration="1.869651837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:11.867440586 +0000 UTC m=+2037.816096425" watchObservedRunningTime="2026-11-02 20:02:11.869651837 +0000 UTC m=+2037.818307673" Nov 02 20:02:12.775735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:12.775697 4966 generic.go:356] "Generic (PLEG): container finished" podID="27652d1d-d665-4e95-b9b5-b231cb1a5202" containerID="0e0684ab7e042dbc7ce1d43cc6d29bb32729b4d11eb870785a64f4430af011b0" exitCode=0 Nov 02 20:02:12.775735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:12.775740 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" event={"ID":"27652d1d-d665-4e95-b9b5-b231cb1a5202","Type":"ContainerDied","Data":"0e0684ab7e042dbc7ce1d43cc6d29bb32729b4d11eb870785a64f4430af011b0"} Nov 02 20:02:13.989351 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:13.989321 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:14.052369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.050728 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27652d1d-d665-4e95-b9b5-b231cb1a5202-kube-api-access-8fj2q\" (UniqueName: \"kubernetes.io/projected/27652d1d-d665-4e95-b9b5-b231cb1a5202-kube-api-access-8fj2q\") pod \"27652d1d-d665-4e95-b9b5-b231cb1a5202\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " Nov 02 20:02:14.052369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.050822 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-xks-ca\") pod \"27652d1d-d665-4e95-b9b5-b231cb1a5202\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " Nov 02 20:02:14.052369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.050852 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-openshift-ca\") pod \"27652d1d-d665-4e95-b9b5-b231cb1a5202\" (UID: \"27652d1d-d665-4e95-b9b5-b231cb1a5202\") " Nov 02 20:02:14.052369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.051340 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-openshift-ca" pod "27652d1d-d665-4e95-b9b5-b231cb1a5202" (UID: "27652d1d-d665-4e95-b9b5-b231cb1a5202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:02:14.052369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.051619 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:14.053354 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.053269 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-xks-ca" pod "27652d1d-d665-4e95-b9b5-b231cb1a5202" (UID: "27652d1d-d665-4e95-b9b5-b231cb1a5202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:02:14.053504 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.053405 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27652d1d-d665-4e95-b9b5-b231cb1a5202-kube-api-access-8fj2q" pod "27652d1d-d665-4e95-b9b5-b231cb1a5202" (UID: "27652d1d-d665-4e95-b9b5-b231cb1a5202"). InnerVolumeSpecName "kube-api-access-8fj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:14.153884 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.153847 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fj2q\" (UniqueName: \"kubernetes.io/projected/27652d1d-d665-4e95-b9b5-b231cb1a5202-kube-api-access-8fj2q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:14.153884 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.153885 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27652d1d-d665-4e95-b9b5-b231cb1a5202-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:14.754642 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:02:14.754598 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:02:14.797167 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.797137 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" Nov 02 20:02:14.797935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.797178 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt" event={"ID":"27652d1d-d665-4e95-b9b5-b231cb1a5202","Type":"ContainerDied","Data":"23966880dbdd75cdc68e307906773d531a20dcfb1c4ffa5f23ce78631c78c20a"} Nov 02 20:02:14.797935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:14.797241 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23966880dbdd75cdc68e307906773d531a20dcfb1c4ffa5f23ce78631c78c20a" Nov 02 20:02:18.205770 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:18.205737 4966 scope.go:122] "RemoveContainer" containerID="ca4b693cfb6cdd0c6a319624df2b56730d101d102ae6ff1ddeddb835e43205d5" Nov 02 20:02:18.216708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:18.216682 4966 scope.go:122] "RemoveContainer" containerID="dffc9b568839e325383c9bd19fc2fe6aa8969933661d44ab03d77bc4a0bc319e" Nov 02 20:02:21.691580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:21.691541 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs"] Nov 02 20:02:21.808466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:21.808433 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf"] Nov 02 20:02:26.692534 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:26.692488 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" podUID="e9f6e754-f50e-4df5-b53b-995aeb4cad39" containerName="payload-processing" containerID="cri-o://c85a34867184ccfa12ac026c6ad4546406435598ba7fe9235bb10b171ca98000" gracePeriod=25 Nov 02 20:02:26.808865 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:26.808829 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" podUID="c4751352-2544-4c5f-b8ef-9f85e50c899f" containerName="payload-pre-processing" containerID="cri-o://ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f" gracePeriod=25 Nov 02 20:02:26.899639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:26.899608 4966 generic.go:356] "Generic (PLEG): container finished" podID="e9f6e754-f50e-4df5-b53b-995aeb4cad39" containerID="c85a34867184ccfa12ac026c6ad4546406435598ba7fe9235bb10b171ca98000" exitCode=0 Nov 02 20:02:26.899790 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:26.899684 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" event={"ID":"e9f6e754-f50e-4df5-b53b-995aeb4cad39","Type":"ContainerDied","Data":"c85a34867184ccfa12ac026c6ad4546406435598ba7fe9235bb10b171ca98000"} Nov 02 20:02:26.978669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:26.978640 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 20:02:27.056466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.051663 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f6e754-f50e-4df5-b53b-995aeb4cad39-kube-api-access-9cshj\" (UniqueName: \"kubernetes.io/projected/e9f6e754-f50e-4df5-b53b-995aeb4cad39-kube-api-access-9cshj\") pod \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " Nov 02 20:02:27.056466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.051966 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f6e754-f50e-4df5-b53b-995aeb4cad39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f6e754-f50e-4df5-b53b-995aeb4cad39-plugins-config-volume\") pod \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\" (UID: \"e9f6e754-f50e-4df5-b53b-995aeb4cad39\") " Nov 02 20:02:27.056466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.053021 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f6e754-f50e-4df5-b53b-995aeb4cad39-plugins-config-volume" pod "e9f6e754-f50e-4df5-b53b-995aeb4cad39" (UID: "e9f6e754-f50e-4df5-b53b-995aeb4cad39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:02:27.069731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.068234 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f6e754-f50e-4df5-b53b-995aeb4cad39-kube-api-access-9cshj" pod "e9f6e754-f50e-4df5-b53b-995aeb4cad39" (UID: "e9f6e754-f50e-4df5-b53b-995aeb4cad39"). InnerVolumeSpecName "kube-api-access-9cshj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:27.123047 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.123016 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 20:02:27.154401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.154358 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f6e754-f50e-4df5-b53b-995aeb4cad39-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:27.154401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.154397 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cshj\" (UniqueName: \"kubernetes.io/projected/e9f6e754-f50e-4df5-b53b-995aeb4cad39-kube-api-access-9cshj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:27.255724 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.255600 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4751352-2544-4c5f-b8ef-9f85e50c899f-kube-api-access-tsc9b\" (UniqueName: \"kubernetes.io/projected/c4751352-2544-4c5f-b8ef-9f85e50c899f-kube-api-access-tsc9b\") pod \"c4751352-2544-4c5f-b8ef-9f85e50c899f\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " Nov 02 20:02:27.256241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.256212 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4751352-2544-4c5f-b8ef-9f85e50c899f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4751352-2544-4c5f-b8ef-9f85e50c899f-plugins-config-volume\") pod \"c4751352-2544-4c5f-b8ef-9f85e50c899f\" (UID: \"c4751352-2544-4c5f-b8ef-9f85e50c899f\") " Nov 02 20:02:27.257776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.257743 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4751352-2544-4c5f-b8ef-9f85e50c899f-plugins-config-volume" pod "c4751352-2544-4c5f-b8ef-9f85e50c899f" (UID: "c4751352-2544-4c5f-b8ef-9f85e50c899f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:02:27.258451 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.258422 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4751352-2544-4c5f-b8ef-9f85e50c899f-kube-api-access-tsc9b" pod "c4751352-2544-4c5f-b8ef-9f85e50c899f" (UID: "c4751352-2544-4c5f-b8ef-9f85e50c899f"). InnerVolumeSpecName "kube-api-access-tsc9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:27.358850 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.358809 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4751352-2544-4c5f-b8ef-9f85e50c899f-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:27.358850 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.358847 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsc9b\" (UniqueName: \"kubernetes.io/projected/c4751352-2544-4c5f-b8ef-9f85e50c899f-kube-api-access-tsc9b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:27.909195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.909160 4966 generic.go:356] "Generic (PLEG): container finished" podID="c4751352-2544-4c5f-b8ef-9f85e50c899f" containerID="ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f" exitCode=0 Nov 02 20:02:27.909676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.909206 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" Nov 02 20:02:27.909676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.909246 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" event={"ID":"c4751352-2544-4c5f-b8ef-9f85e50c899f","Type":"ContainerDied","Data":"ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f"} Nov 02 20:02:27.909676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.909314 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf" event={"ID":"c4751352-2544-4c5f-b8ef-9f85e50c899f","Type":"ContainerDied","Data":"86f60049e41b42526b698545ba294caa375a103cd8953ea48d5ef19fa453d804"} Nov 02 20:02:27.909676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.909347 4966 scope.go:122] "RemoveContainer" containerID="ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f" Nov 02 20:02:27.910858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.910833 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" event={"ID":"e9f6e754-f50e-4df5-b53b-995aeb4cad39","Type":"ContainerDied","Data":"b15dea58bc7fa36ed7ded4359d8a28635a4b5aec61a7d94d26062736139e27d3"} Nov 02 20:02:27.910858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.910843 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs" Nov 02 20:02:27.921239 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.921214 4966 scope.go:122] "RemoveContainer" containerID="ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f" Nov 02 20:02:27.921541 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:02:27.921522 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f\": container with ID starting with ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f not found: ID does not exist" containerID="ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f" Nov 02 20:02:27.921622 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.921549 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f"} err="failed to get container status \"ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f\": rpc error: code = NotFound desc = could not find container \"ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f\": container with ID starting with ef047d7c0fb32a3acab13427e3d7b2902bccd8208c7cdf9311df57b69198161f not found: ID does not exist" Nov 02 20:02:27.921622 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:27.921567 4966 scope.go:122] "RemoveContainer" containerID="c85a34867184ccfa12ac026c6ad4546406435598ba7fe9235bb10b171ca98000" Nov 02 20:02:28.080978 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:28.080938 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf"] Nov 02 20:02:28.124120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:28.124077 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b0cc24-76f6b7675-5jrrf"] Nov 02 20:02:28.198728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:28.198697 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4751352-2544-4c5f-b8ef-9f85e50c899f" path="/var/lib/kubelet/pods/c4751352-2544-4c5f-b8ef-9f85e50c899f/volumes" Nov 02 20:02:28.199099 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:28.199079 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs"] Nov 02 20:02:28.199140 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:28.199120 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b0cc24-58d8c559b5-rgxgs"] Nov 02 20:02:30.184591 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:30.184560 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f6e754-f50e-4df5-b53b-995aeb4cad39" path="/var/lib/kubelet/pods/e9f6e754-f50e-4df5-b53b-995aeb4cad39/volumes" Nov 02 20:02:44.754338 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:02:44.754273 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:02:47.456887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:47.456848 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw"] Nov 02 20:02:47.509930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:47.509893 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8479ff-kwwkw"] Nov 02 20:02:48.186220 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:48.186187 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edbc517-0b2b-4c36-ab24-44bc97e86b38" path="/var/lib/kubelet/pods/5edbc517-0b2b-4c36-ab24-44bc97e86b38/volumes" Nov 02 20:02:56.066735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.066700 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs"] Nov 02 20:02:56.067420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067402 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e9f6e754-f50e-4df5-b53b-995aeb4cad39" containerName="payload-processing" Nov 02 20:02:56.067420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067420 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f6e754-f50e-4df5-b53b-995aeb4cad39" containerName="payload-processing" Nov 02 20:02:56.067574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067434 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c4751352-2544-4c5f-b8ef-9f85e50c899f" containerName="payload-pre-processing" Nov 02 20:02:56.067574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067439 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4751352-2544-4c5f-b8ef-9f85e50c899f" containerName="payload-pre-processing" Nov 02 20:02:56.067574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067463 4966 cpu_manager.go:414] "containerMap: removing container" podUID="27652d1d-d665-4e95-b9b5-b231cb1a5202" containerName="revoke-keys" Nov 02 20:02:56.067574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067469 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27652d1d-d665-4e95-b9b5-b231cb1a5202" containerName="revoke-keys" Nov 02 20:02:56.067739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067585 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f6e754-f50e-4df5-b53b-995aeb4cad39" containerName="payload-processing" Nov 02 20:02:56.067739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067594 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4751352-2544-4c5f-b8ef-9f85e50c899f" containerName="payload-pre-processing" Nov 02 20:02:56.067739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.067605 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="27652d1d-d665-4e95-b9b5-b231cb1a5202" containerName="revoke-keys" Nov 02 20:02:56.072412 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.072391 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.124004 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.123975 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs"] Nov 02 20:02:56.165131 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.165094 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:02:56.166568 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.166552 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:02:56.166686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.166580 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:02:56.166686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.166583 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:02:56.178508 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.178393 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24kr\" (UniqueName: \"kubernetes.io/projected/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-kube-api-access-m24kr\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.178508 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.178488 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.178690 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.178609 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.281638 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.281601 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24kr\" (UniqueName: \"kubernetes.io/projected/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-kube-api-access-m24kr\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.282023 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.281998 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.282326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.282286 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.285732 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.285697 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.287861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.287831 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.339117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.339037 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24kr\" (UniqueName: \"kubernetes.io/projected/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-kube-api-access-m24kr\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.386838 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.386803 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:56.765352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.765317 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs"] Nov 02 20:02:56.765584 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:02:56.765446 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb8a738_05b9_4350_a5f5_5be8ac3ffd0c.slice/crio-55a8248d118dab30359fde2e5f489e81fb3caf5b42ea21a19f441f101af7784b WatchSource:0}: Error finding container 55a8248d118dab30359fde2e5f489e81fb3caf5b42ea21a19f441f101af7784b: Status 404 returned error can't find the container with id 55a8248d118dab30359fde2e5f489e81fb3caf5b42ea21a19f441f101af7784b Nov 02 20:02:56.785709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.785662 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw"] Nov 02 20:02:56.793091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.793055 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.825731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.825703 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw"] Nov 02 20:02:56.890713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.890674 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r97z\" (UniqueName: \"kubernetes.io/projected/c8fd7003-0102-48f2-94eb-ba8e542eec43-kube-api-access-6r97z\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.890890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.890734 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.890890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.890793 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.992195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.992161 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.992442 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.992200 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.992442 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.992426 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r97z\" (UniqueName: \"kubernetes.io/projected/c8fd7003-0102-48f2-94eb-ba8e542eec43-kube-api-access-6r97z\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.992907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.992882 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:56.994723 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:56.994696 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:57.044977 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:57.044917 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r97z\" (UniqueName: \"kubernetes.io/projected/c8fd7003-0102-48f2-94eb-ba8e542eec43-kube-api-access-6r97z\") pod \"maas-api-revoke-keys-e2e-mt-7ab11604-slkxw\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:57.124647 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:57.124622 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:02:57.200796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:57.200752 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" event={"ID":"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c","Type":"ContainerStarted","Data":"55a8248d118dab30359fde2e5f489e81fb3caf5b42ea21a19f441f101af7784b"} Nov 02 20:02:57.497952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:57.497921 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw"] Nov 02 20:02:57.498099 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:02:57.497963 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fd7003_0102_48f2_94eb_ba8e542eec43.slice/crio-e8da99d98a8abfd3c959ef98d3f8c37bb93b75e1b54d29021ac663c1143ae3be WatchSource:0}: Error finding container e8da99d98a8abfd3c959ef98d3f8c37bb93b75e1b54d29021ac663c1143ae3be: Status 404 returned error can't find the container with id e8da99d98a8abfd3c959ef98d3f8c37bb93b75e1b54d29021ac663c1143ae3be Nov 02 20:02:58.211659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:58.211556 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" event={"ID":"c8fd7003-0102-48f2-94eb-ba8e542eec43","Type":"ContainerStarted","Data":"20e4fd80a95dfc190668e70b2bcfe297640221eb6773ac6a74c1fe74da720955"} Nov 02 20:02:58.211659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:58.211619 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" event={"ID":"c8fd7003-0102-48f2-94eb-ba8e542eec43","Type":"ContainerStarted","Data":"e8da99d98a8abfd3c959ef98d3f8c37bb93b75e1b54d29021ac663c1143ae3be"} Nov 02 20:02:58.213339 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:58.213287 4966 generic.go:356] "Generic (PLEG): container finished" podID="0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" containerID="b6e9eae49bdc6c9dc5000e4fbcd572e388252ab2b445dc56873718ed4d4ad864" exitCode=0 Nov 02 20:02:58.213465 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:58.213321 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" event={"ID":"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c","Type":"ContainerDied","Data":"b6e9eae49bdc6c9dc5000e4fbcd572e388252ab2b445dc56873718ed4d4ad864"} Nov 02 20:02:58.320495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:58.320447 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" podStartSLOduration=2.3204337600000002 podStartE2EDuration="2.32043376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:58.315389322 +0000 UTC m=+2084.264045161" watchObservedRunningTime="2026-11-02 20:02:58.32043376 +0000 UTC m=+2084.269089599" Nov 02 20:02:59.231108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.231060 4966 generic.go:356] "Generic (PLEG): container finished" podID="c8fd7003-0102-48f2-94eb-ba8e542eec43" containerID="20e4fd80a95dfc190668e70b2bcfe297640221eb6773ac6a74c1fe74da720955" exitCode=0 Nov 02 20:02:59.231582 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.231141 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" event={"ID":"c8fd7003-0102-48f2-94eb-ba8e542eec43","Type":"ContainerDied","Data":"20e4fd80a95dfc190668e70b2bcfe297640221eb6773ac6a74c1fe74da720955"} Nov 02 20:02:59.414928 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.414891 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:02:59.533052 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.532968 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-kube-api-access-m24kr\" (UniqueName: \"kubernetes.io/projected/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-kube-api-access-m24kr\") pod \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " Nov 02 20:02:59.533234 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.533149 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-xks-ca\") pod \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " Nov 02 20:02:59.533317 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.533265 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-openshift-ca\") pod \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\" (UID: \"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c\") " Nov 02 20:02:59.533808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.533770 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-openshift-ca" pod "0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" (UID: "0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:02:59.535535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.535502 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-xks-ca" pod "0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" (UID: "0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:02:59.535660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.535637 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-kube-api-access-m24kr" pod "0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" (UID: "0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c"). InnerVolumeSpecName "kube-api-access-m24kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:59.635751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.635706 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:59.635751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.635745 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m24kr\" (UniqueName: \"kubernetes.io/projected/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-kube-api-access-m24kr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:59.635751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:02:59.635758 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:00.242171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.242140 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" Nov 02 20:03:00.242669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.242173 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs" event={"ID":"0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c","Type":"ContainerDied","Data":"55a8248d118dab30359fde2e5f489e81fb3caf5b42ea21a19f441f101af7784b"} Nov 02 20:03:00.242669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.242229 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55a8248d118dab30359fde2e5f489e81fb3caf5b42ea21a19f441f101af7784b" Nov 02 20:03:00.417888 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.417861 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:03:00.447106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.447032 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-openshift-ca\") pod \"c8fd7003-0102-48f2-94eb-ba8e542eec43\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " Nov 02 20:03:00.447106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.447102 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fd7003-0102-48f2-94eb-ba8e542eec43-kube-api-access-6r97z\" (UniqueName: \"kubernetes.io/projected/c8fd7003-0102-48f2-94eb-ba8e542eec43-kube-api-access-6r97z\") pod \"c8fd7003-0102-48f2-94eb-ba8e542eec43\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " Nov 02 20:03:00.447288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.447187 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-xks-ca\") pod \"c8fd7003-0102-48f2-94eb-ba8e542eec43\" (UID: \"c8fd7003-0102-48f2-94eb-ba8e542eec43\") " Nov 02 20:03:00.447436 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.447412 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-openshift-ca" pod "c8fd7003-0102-48f2-94eb-ba8e542eec43" (UID: "c8fd7003-0102-48f2-94eb-ba8e542eec43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:00.447833 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.447809 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:00.449388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.449359 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fd7003-0102-48f2-94eb-ba8e542eec43-kube-api-access-6r97z" pod "c8fd7003-0102-48f2-94eb-ba8e542eec43" (UID: "c8fd7003-0102-48f2-94eb-ba8e542eec43"). InnerVolumeSpecName "kube-api-access-6r97z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:00.449486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.449365 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-xks-ca" pod "c8fd7003-0102-48f2-94eb-ba8e542eec43" (UID: "c8fd7003-0102-48f2-94eb-ba8e542eec43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:00.548733 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.548699 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fd7003-0102-48f2-94eb-ba8e542eec43-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:00.548733 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.548730 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r97z\" (UniqueName: \"kubernetes.io/projected/c8fd7003-0102-48f2-94eb-ba8e542eec43-kube-api-access-6r97z\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:00.575357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.575328 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6"] Nov 02 20:03:00.575991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.575977 4966 cpu_manager.go:414] "containerMap: removing container" podUID="0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" containerName="revoke-keys" Nov 02 20:03:00.576050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.575991 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" containerName="revoke-keys" Nov 02 20:03:00.576050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.576013 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c8fd7003-0102-48f2-94eb-ba8e542eec43" containerName="revoke-keys" Nov 02 20:03:00.576050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.576019 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fd7003-0102-48f2-94eb-ba8e542eec43" containerName="revoke-keys" Nov 02 20:03:00.576165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.576143 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" containerName="revoke-keys" Nov 02 20:03:00.576165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.576156 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fd7003-0102-48f2-94eb-ba8e542eec43" containerName="revoke-keys" Nov 02 20:03:00.579837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.579816 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.632960 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.632930 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6"] Nov 02 20:03:00.650642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.650606 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv72w\" (UniqueName: \"kubernetes.io/projected/8a527bdb-0065-4f8c-85e2-77b2da213975-kube-api-access-sv72w\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.650851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.650824 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.651089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.651067 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.752201 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.752160 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.752477 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.752233 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv72w\" (UniqueName: \"kubernetes.io/projected/8a527bdb-0065-4f8c-85e2-77b2da213975-kube-api-access-sv72w\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.752477 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.752370 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.752983 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.752955 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.754791 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.754768 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.805209 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.805168 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv72w\" (UniqueName: \"kubernetes.io/projected/8a527bdb-0065-4f8c-85e2-77b2da213975-kube-api-access-sv72w\") pod \"maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:00.901268 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:00.901228 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:01.251448 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:01.251422 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" Nov 02 20:03:01.251845 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:01.251452 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw" event={"ID":"c8fd7003-0102-48f2-94eb-ba8e542eec43","Type":"ContainerDied","Data":"e8da99d98a8abfd3c959ef98d3f8c37bb93b75e1b54d29021ac663c1143ae3be"} Nov 02 20:03:01.251845 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:01.251506 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8da99d98a8abfd3c959ef98d3f8c37bb93b75e1b54d29021ac663c1143ae3be" Nov 02 20:03:01.301539 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:01.301504 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6"] Nov 02 20:03:01.301852 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:01.301819 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a527bdb_0065_4f8c_85e2_77b2da213975.slice/crio-490c8c2c76dce5309c46676ce2d53b0d26930d03d86c9005cc4ffd2ba114a1b0 WatchSource:0}: Error finding container 490c8c2c76dce5309c46676ce2d53b0d26930d03d86c9005cc4ffd2ba114a1b0: Status 404 returned error can't find the container with id 490c8c2c76dce5309c46676ce2d53b0d26930d03d86c9005cc4ffd2ba114a1b0 Nov 02 20:03:02.262333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:02.262270 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" event={"ID":"8a527bdb-0065-4f8c-85e2-77b2da213975","Type":"ContainerStarted","Data":"a47dd1d3225335b5272fd1968026cb851f857a3de85ee42572669f3344e425e8"} Nov 02 20:03:02.262846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:02.262348 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" event={"ID":"8a527bdb-0065-4f8c-85e2-77b2da213975","Type":"ContainerStarted","Data":"490c8c2c76dce5309c46676ce2d53b0d26930d03d86c9005cc4ffd2ba114a1b0"} Nov 02 20:03:02.368634 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:02.368564 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" podStartSLOduration=2.3685405299999998 podStartE2EDuration="2.36854053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:02.366602989 +0000 UTC m=+2088.315258823" watchObservedRunningTime="2026-11-02 20:03:02.36854053 +0000 UTC m=+2088.317196364" Nov 02 20:03:03.274288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:03.274253 4966 generic.go:356] "Generic (PLEG): container finished" podID="8a527bdb-0065-4f8c-85e2-77b2da213975" containerID="a47dd1d3225335b5272fd1968026cb851f857a3de85ee42572669f3344e425e8" exitCode=0 Nov 02 20:03:03.274708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:03.274330 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" event={"ID":"8a527bdb-0065-4f8c-85e2-77b2da213975","Type":"ContainerDied","Data":"a47dd1d3225335b5272fd1968026cb851f857a3de85ee42572669f3344e425e8"} Nov 02 20:03:04.461783 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.461757 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:04.603327 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.603226 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-openshift-ca\") pod \"8a527bdb-0065-4f8c-85e2-77b2da213975\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " Nov 02 20:03:04.603519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.603340 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-xks-ca\") pod \"8a527bdb-0065-4f8c-85e2-77b2da213975\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " Nov 02 20:03:04.603519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.603500 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a527bdb-0065-4f8c-85e2-77b2da213975-kube-api-access-sv72w\" (UniqueName: \"kubernetes.io/projected/8a527bdb-0065-4f8c-85e2-77b2da213975-kube-api-access-sv72w\") pod \"8a527bdb-0065-4f8c-85e2-77b2da213975\" (UID: \"8a527bdb-0065-4f8c-85e2-77b2da213975\") " Nov 02 20:03:04.603709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.603679 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-openshift-ca" pod "8a527bdb-0065-4f8c-85e2-77b2da213975" (UID: "8a527bdb-0065-4f8c-85e2-77b2da213975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:04.604096 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.604064 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:04.605701 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.605675 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a527bdb-0065-4f8c-85e2-77b2da213975-kube-api-access-sv72w" pod "8a527bdb-0065-4f8c-85e2-77b2da213975" (UID: "8a527bdb-0065-4f8c-85e2-77b2da213975"). InnerVolumeSpecName "kube-api-access-sv72w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:04.605859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.605834 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-xks-ca" pod "8a527bdb-0065-4f8c-85e2-77b2da213975" (UID: "8a527bdb-0065-4f8c-85e2-77b2da213975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:04.705289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.705250 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv72w\" (UniqueName: \"kubernetes.io/projected/8a527bdb-0065-4f8c-85e2-77b2da213975-kube-api-access-sv72w\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:04.705289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:04.705286 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a527bdb-0065-4f8c-85e2-77b2da213975-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:05.293090 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:05.293057 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" Nov 02 20:03:05.293090 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:05.293084 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6" event={"ID":"8a527bdb-0065-4f8c-85e2-77b2da213975","Type":"ContainerDied","Data":"490c8c2c76dce5309c46676ce2d53b0d26930d03d86c9005cc4ffd2ba114a1b0"} Nov 02 20:03:05.293376 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:05.293135 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490c8c2c76dce5309c46676ce2d53b0d26930d03d86c9005cc4ffd2ba114a1b0" Nov 02 20:03:07.595921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:07.595879 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9"] Nov 02 20:03:07.712149 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:07.712113 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f"] Nov 02 20:03:11.047533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:11.047485 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn"] Nov 02 20:03:11.092698 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:11.092666 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e365778-hggqn"] Nov 02 20:03:11.212948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:11.212905 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk"] Nov 02 20:03:11.216934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:11.216910 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-23a9f5-s2vbk"] Nov 02 20:03:12.190482 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:12.190435 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd23151-254b-4588-ab5f-03d658975eb3" path="/var/lib/kubelet/pods/0bd23151-254b-4588-ab5f-03d658975eb3/volumes" Nov 02 20:03:12.191115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:12.191095 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca75052c-7bda-4920-8b53-4cd06691abc9" path="/var/lib/kubelet/pods/ca75052c-7bda-4920-8b53-4cd06691abc9/volumes" Nov 02 20:03:12.597270 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:12.597219 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" podUID="4ba52fba-eeac-4146-9792-a30af53544ee" containerName="payload-processing" containerID="cri-o://afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558" gracePeriod=25 Nov 02 20:03:12.713013 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:12.712960 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" podUID="27e043f8-4ed5-423d-912a-d7b3bdeb328f" containerName="payload-pre-processing" containerID="cri-o://ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa" gracePeriod=25 Nov 02 20:03:12.916214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:12.916190 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 20:03:12.989377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:12.989345 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 20:03:13.014672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.014641 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl"] Nov 02 20:03:13.014815 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.014697 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e043f8-4ed5-423d-912a-d7b3bdeb328f-kube-api-access-4mnld\" (UniqueName: \"kubernetes.io/projected/27e043f8-4ed5-423d-912a-d7b3bdeb328f-kube-api-access-4mnld\") pod \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " Nov 02 20:03:13.014815 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.014796 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba52fba-eeac-4146-9792-a30af53544ee-kube-api-access-b8vvc\" (UniqueName: \"kubernetes.io/projected/4ba52fba-eeac-4146-9792-a30af53544ee-kube-api-access-b8vvc\") pod \"4ba52fba-eeac-4146-9792-a30af53544ee\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " Nov 02 20:03:13.015052 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.015012 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba52fba-eeac-4146-9792-a30af53544ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba52fba-eeac-4146-9792-a30af53544ee-plugins-config-volume\") pod \"4ba52fba-eeac-4146-9792-a30af53544ee\" (UID: \"4ba52fba-eeac-4146-9792-a30af53544ee\") " Nov 02 20:03:13.015146 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.015092 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e043f8-4ed5-423d-912a-d7b3bdeb328f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e043f8-4ed5-423d-912a-d7b3bdeb328f-plugins-config-volume\") pod \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\" (UID: \"27e043f8-4ed5-423d-912a-d7b3bdeb328f\") " Nov 02 20:03:13.015647 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.015625 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e043f8-4ed5-423d-912a-d7b3bdeb328f-plugins-config-volume" pod "27e043f8-4ed5-423d-912a-d7b3bdeb328f" (UID: "27e043f8-4ed5-423d-912a-d7b3bdeb328f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:13.015770 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.015684 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba52fba-eeac-4146-9792-a30af53544ee-plugins-config-volume" pod "4ba52fba-eeac-4146-9792-a30af53544ee" (UID: "4ba52fba-eeac-4146-9792-a30af53544ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:13.017335 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.017273 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e043f8-4ed5-423d-912a-d7b3bdeb328f-kube-api-access-4mnld" pod "27e043f8-4ed5-423d-912a-d7b3bdeb328f" (UID: "27e043f8-4ed5-423d-912a-d7b3bdeb328f"). InnerVolumeSpecName "kube-api-access-4mnld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:13.017427 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.017372 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba52fba-eeac-4146-9792-a30af53544ee-kube-api-access-b8vvc" pod "4ba52fba-eeac-4146-9792-a30af53544ee" (UID: "4ba52fba-eeac-4146-9792-a30af53544ee"). InnerVolumeSpecName "kube-api-access-b8vvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:13.112059 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.112024 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82"] Nov 02 20:03:13.117620 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.117589 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba52fba-eeac-4146-9792-a30af53544ee-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:13.117620 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.117616 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e043f8-4ed5-423d-912a-d7b3bdeb328f-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:13.117862 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.117626 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mnld\" (UniqueName: \"kubernetes.io/projected/27e043f8-4ed5-423d-912a-d7b3bdeb328f-kube-api-access-4mnld\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:13.117862 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.117640 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vvc\" (UniqueName: \"kubernetes.io/projected/4ba52fba-eeac-4146-9792-a30af53544ee-kube-api-access-b8vvc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:13.404122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.404089 4966 generic.go:356] "Generic (PLEG): container finished" podID="4ba52fba-eeac-4146-9792-a30af53544ee" containerID="afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558" exitCode=0 Nov 02 20:03:13.404686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.404135 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" Nov 02 20:03:13.404686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.404184 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" event={"ID":"4ba52fba-eeac-4146-9792-a30af53544ee","Type":"ContainerDied","Data":"afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558"} Nov 02 20:03:13.404686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.404248 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9" event={"ID":"4ba52fba-eeac-4146-9792-a30af53544ee","Type":"ContainerDied","Data":"f4808dab65170806885916cb63ba6ce2c1f13fcd23d18537787d5cce1fd7f0e2"} Nov 02 20:03:13.404686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.404279 4966 scope.go:122] "RemoveContainer" containerID="afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558" Nov 02 20:03:13.406181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.406153 4966 generic.go:356] "Generic (PLEG): container finished" podID="27e043f8-4ed5-423d-912a-d7b3bdeb328f" containerID="ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa" exitCode=0 Nov 02 20:03:13.406492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.406469 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" event={"ID":"27e043f8-4ed5-423d-912a-d7b3bdeb328f","Type":"ContainerDied","Data":"ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa"} Nov 02 20:03:13.406552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.406521 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" event={"ID":"27e043f8-4ed5-423d-912a-d7b3bdeb328f","Type":"ContainerDied","Data":"0d55f7ac313f652beff5a70318d512bd163fe3a88f89c2715741aa8a2caac77f"} Nov 02 20:03:13.406596 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.406587 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f" Nov 02 20:03:13.416104 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.416085 4966 scope.go:122] "RemoveContainer" containerID="afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558" Nov 02 20:03:13.416419 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:03:13.416396 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558\": container with ID starting with afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558 not found: ID does not exist" containerID="afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558" Nov 02 20:03:13.416507 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.416429 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558"} err="failed to get container status \"afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558\": rpc error: code = NotFound desc = could not find container \"afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558\": container with ID starting with afe8dacf07f7e1a5750b8b712391104ca8b45259f8fc21048e472a1b8785d558 not found: ID does not exist" Nov 02 20:03:13.416507 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.416453 4966 scope.go:122] "RemoveContainer" containerID="ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa" Nov 02 20:03:13.430459 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.429984 4966 scope.go:122] "RemoveContainer" containerID="ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa" Nov 02 20:03:13.430459 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:03:13.430355 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa\": container with ID starting with ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa not found: ID does not exist" containerID="ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa" Nov 02 20:03:13.430459 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.430393 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa"} err="failed to get container status \"ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa\": rpc error: code = NotFound desc = could not find container \"ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa\": container with ID starting with ac1afa476e545cebe2646c8863100621970f5b5e3bbdd731d30066a7dcf82cfa not found: ID does not exist" Nov 02 20:03:13.564276 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.564241 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9"] Nov 02 20:03:13.569126 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.569090 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ab11604-6d8c84d59f-5dkp9"] Nov 02 20:03:13.673061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.672969 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f"] Nov 02 20:03:13.677233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:13.677203 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ab11604-56f9b7469f-5fq2f"] Nov 02 20:03:14.188812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:14.188780 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e043f8-4ed5-423d-912a-d7b3bdeb328f" path="/var/lib/kubelet/pods/27e043f8-4ed5-423d-912a-d7b3bdeb328f/volumes" Nov 02 20:03:14.189223 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:14.189201 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba52fba-eeac-4146-9792-a30af53544ee" path="/var/lib/kubelet/pods/4ba52fba-eeac-4146-9792-a30af53544ee/volumes" Nov 02 20:03:14.766336 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:03:14.766269 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:03:14.920596 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:14.920556 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:03:14.936026 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:14.935997 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:03:14.938784 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:14.938762 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:03:14.956902 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:14.956872 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:03:17.061406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:17.061375 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw"] Nov 02 20:03:17.106393 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:17.106357 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7ab11604-slkxw"] Nov 02 20:03:18.023337 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.017403 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" podUID="029571de-57d3-4dca-91f2-22fd2df6a513" containerName="payload-processing" containerID="cri-o://34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a" gracePeriod=25 Nov 02 20:03:18.116866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.112636 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" podUID="9f5c6025-a0af-455b-80b6-b38225915f4c" containerName="payload-pre-processing" containerID="cri-o://9f0fa0f11640727bd6fd861a968c6e3ac25e68fa1be7fbdce0fc8f7f1fa23e64" gracePeriod=25 Nov 02 20:03:18.236990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.236959 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fd7003-0102-48f2-94eb-ba8e542eec43" path="/var/lib/kubelet/pods/c8fd7003-0102-48f2-94eb-ba8e542eec43/volumes" Nov 02 20:03:18.248288 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:18.246740 4966 logging.go:55] [core] [Channel #4038 SubChannel #4040]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.203:50052", ServerName: "10.132.0.203:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.203:50052: connect: connection refused" Nov 02 20:03:18.248288 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:18.246740 4966 logging.go:55] [core] [Channel #4037 SubChannel #4039]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.202:50052", ServerName: "10.132.0.202:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.202:50052: connect: connection refused" Nov 02 20:03:18.304137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.304035 4966 scope.go:122] "RemoveContainer" containerID="ec3664489993a01f6c23f630b2630d4d4dec6e599439625cbfc5033683602b98" Nov 02 20:03:18.399111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.399081 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:03:18.452518 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.452481 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029571de-57d3-4dca-91f2-22fd2df6a513-kube-api-access-pfr9q\" (UniqueName: \"kubernetes.io/projected/029571de-57d3-4dca-91f2-22fd2df6a513-kube-api-access-pfr9q\") pod \"029571de-57d3-4dca-91f2-22fd2df6a513\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " Nov 02 20:03:18.452819 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.452800 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029571de-57d3-4dca-91f2-22fd2df6a513-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029571de-57d3-4dca-91f2-22fd2df6a513-plugins-config-volume\") pod \"029571de-57d3-4dca-91f2-22fd2df6a513\" (UID: \"029571de-57d3-4dca-91f2-22fd2df6a513\") " Nov 02 20:03:18.453154 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.453113 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029571de-57d3-4dca-91f2-22fd2df6a513-plugins-config-volume" pod "029571de-57d3-4dca-91f2-22fd2df6a513" (UID: "029571de-57d3-4dca-91f2-22fd2df6a513"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:18.453290 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.453270 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029571de-57d3-4dca-91f2-22fd2df6a513-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:18.455113 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.455084 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029571de-57d3-4dca-91f2-22fd2df6a513-kube-api-access-pfr9q" pod "029571de-57d3-4dca-91f2-22fd2df6a513" (UID: "029571de-57d3-4dca-91f2-22fd2df6a513"). InnerVolumeSpecName "kube-api-access-pfr9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:18.484142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.484116 4966 generic.go:356] "Generic (PLEG): container finished" podID="029571de-57d3-4dca-91f2-22fd2df6a513" containerID="34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a" exitCode=0 Nov 02 20:03:18.484257 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.484154 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" Nov 02 20:03:18.484257 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.484195 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" event={"ID":"029571de-57d3-4dca-91f2-22fd2df6a513","Type":"ContainerDied","Data":"34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a"} Nov 02 20:03:18.484406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.484280 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" event={"ID":"029571de-57d3-4dca-91f2-22fd2df6a513","Type":"ContainerDied","Data":"682c03ee450d99647b0b667f296a1c84c33daaf11c30b75b106de15a5c7411f3"} Nov 02 20:03:18.484406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.484354 4966 scope.go:122] "RemoveContainer" containerID="34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a" Nov 02 20:03:18.484772 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.484741 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:03:18.486505 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.486477 4966 generic.go:356] "Generic (PLEG): container finished" podID="9f5c6025-a0af-455b-80b6-b38225915f4c" containerID="9f0fa0f11640727bd6fd861a968c6e3ac25e68fa1be7fbdce0fc8f7f1fa23e64" exitCode=0 Nov 02 20:03:18.486631 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.486532 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" event={"ID":"9f5c6025-a0af-455b-80b6-b38225915f4c","Type":"ContainerDied","Data":"9f0fa0f11640727bd6fd861a968c6e3ac25e68fa1be7fbdce0fc8f7f1fa23e64"} Nov 02 20:03:18.496003 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.495979 4966 scope.go:122] "RemoveContainer" containerID="34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a" Nov 02 20:03:18.496333 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:03:18.496287 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a\": container with ID starting with 34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a not found: ID does not exist" containerID="34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a" Nov 02 20:03:18.496428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.496330 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a"} err="failed to get container status \"34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a\": rpc error: code = NotFound desc = could not find container \"34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a\": container with ID starting with 34c1ff11bd6f994a8ed74c1a9a3fa48f73c01a14ca308439ae88054ebf14df2a not found: ID does not exist" Nov 02 20:03:18.496428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.496350 4966 scope.go:122] "RemoveContainer" containerID="9f0fa0f11640727bd6fd861a968c6e3ac25e68fa1be7fbdce0fc8f7f1fa23e64" Nov 02 20:03:18.554113 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.554079 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5c6025-a0af-455b-80b6-b38225915f4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5c6025-a0af-455b-80b6-b38225915f4c-plugins-config-volume\") pod \"9f5c6025-a0af-455b-80b6-b38225915f4c\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " Nov 02 20:03:18.554438 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.554384 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5c6025-a0af-455b-80b6-b38225915f4c-kube-api-access-m886n\" (UniqueName: \"kubernetes.io/projected/9f5c6025-a0af-455b-80b6-b38225915f4c-kube-api-access-m886n\") pod \"9f5c6025-a0af-455b-80b6-b38225915f4c\" (UID: \"9f5c6025-a0af-455b-80b6-b38225915f4c\") " Nov 02 20:03:18.555456 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.555429 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfr9q\" (UniqueName: \"kubernetes.io/projected/029571de-57d3-4dca-91f2-22fd2df6a513-kube-api-access-pfr9q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:18.555634 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.555604 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5c6025-a0af-455b-80b6-b38225915f4c-plugins-config-volume" pod "9f5c6025-a0af-455b-80b6-b38225915f4c" (UID: "9f5c6025-a0af-455b-80b6-b38225915f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:18.557185 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.557158 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5c6025-a0af-455b-80b6-b38225915f4c-kube-api-access-m886n" pod "9f5c6025-a0af-455b-80b6-b38225915f4c" (UID: "9f5c6025-a0af-455b-80b6-b38225915f4c"). InnerVolumeSpecName "kube-api-access-m886n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:18.657244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.657201 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5c6025-a0af-455b-80b6-b38225915f4c-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:18.657244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.657245 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m886n\" (UniqueName: \"kubernetes.io/projected/9f5c6025-a0af-455b-80b6-b38225915f4c-kube-api-access-m886n\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:18.719443 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.719404 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl"] Nov 02 20:03:18.724332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:18.724307 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl"] Nov 02 20:03:19.246950 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:19.246898 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-9f8f23-7d766f8647-qslbl" podUID="029571de-57d3-4dca-91f2-22fd2df6a513" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.203:50052\" within 1s: context deadline exceeded" Nov 02 20:03:19.247361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:19.246923 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" podUID="9f5c6025-a0af-455b-80b6-b38225915f4c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.202:50052\" within 1s: context deadline exceeded" Nov 02 20:03:19.497022 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:19.496989 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" Nov 02 20:03:19.497196 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:19.497024 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82" event={"ID":"9f5c6025-a0af-455b-80b6-b38225915f4c","Type":"ContainerDied","Data":"27afa3ee11eb0637f683b4e74541807b26521fef742fd417625b733048b9d67c"} Nov 02 20:03:19.651887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:19.651828 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82"] Nov 02 20:03:19.657988 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:19.657957 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f8f23-748786477f-mcn82"] Nov 02 20:03:20.188741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:20.188707 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029571de-57d3-4dca-91f2-22fd2df6a513" path="/var/lib/kubelet/pods/029571de-57d3-4dca-91f2-22fd2df6a513/volumes" Nov 02 20:03:20.189142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:20.189128 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5c6025-a0af-455b-80b6-b38225915f4c" path="/var/lib/kubelet/pods/9f5c6025-a0af-455b-80b6-b38225915f4c/volumes" Nov 02 20:03:22.977116 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:22.977077 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6"] Nov 02 20:03:23.022269 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:23.022238 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8f23-76nx6"] Nov 02 20:03:24.186192 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:24.186154 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a527bdb-0065-4f8c-85e2-77b2da213975" path="/var/lib/kubelet/pods/8a527bdb-0065-4f8c-85e2-77b2da213975/volumes" Nov 02 20:03:24.374620 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:24.374588 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh"] Nov 02 20:03:24.378328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:24.378273 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-89a22a-p5zgh"] Nov 02 20:03:26.186415 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:26.186325 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b" path="/var/lib/kubelet/pods/fd888ca0-3e2b-4b3f-8a98-9abaeba31d3b/volumes" Nov 02 20:03:30.049345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.049293 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm"] Nov 02 20:03:30.050623 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050590 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4ba52fba-eeac-4146-9792-a30af53544ee" containerName="payload-processing" Nov 02 20:03:30.050623 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050614 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba52fba-eeac-4146-9792-a30af53544ee" containerName="payload-processing" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050644 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8a527bdb-0065-4f8c-85e2-77b2da213975" containerName="revoke-keys" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050652 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a527bdb-0065-4f8c-85e2-77b2da213975" containerName="revoke-keys" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050674 4966 cpu_manager.go:414] "containerMap: removing container" podUID="9f5c6025-a0af-455b-80b6-b38225915f4c" containerName="payload-pre-processing" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050681 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5c6025-a0af-455b-80b6-b38225915f4c" containerName="payload-pre-processing" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050721 4966 cpu_manager.go:414] "containerMap: removing container" podUID="029571de-57d3-4dca-91f2-22fd2df6a513" containerName="payload-processing" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050728 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029571de-57d3-4dca-91f2-22fd2df6a513" containerName="payload-processing" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050751 4966 cpu_manager.go:414] "containerMap: removing container" podUID="27e043f8-4ed5-423d-912a-d7b3bdeb328f" containerName="payload-pre-processing" Nov 02 20:03:30.050827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050759 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e043f8-4ed5-423d-912a-d7b3bdeb328f" containerName="payload-pre-processing" Nov 02 20:03:30.051248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050965 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a527bdb-0065-4f8c-85e2-77b2da213975" containerName="revoke-keys" Nov 02 20:03:30.051248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.050983 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5c6025-a0af-455b-80b6-b38225915f4c" containerName="payload-pre-processing" Nov 02 20:03:30.051248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.051001 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="029571de-57d3-4dca-91f2-22fd2df6a513" containerName="payload-processing" Nov 02 20:03:30.051248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.051013 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e043f8-4ed5-423d-912a-d7b3bdeb328f" containerName="payload-pre-processing" Nov 02 20:03:30.051248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.051025 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba52fba-eeac-4146-9792-a30af53544ee" containerName="payload-processing" Nov 02 20:03:30.065315 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.063641 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.107516 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.107489 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm"] Nov 02 20:03:30.141319 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.141252 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5b64d-dockercfg-jvk7c\"" Nov 02 20:03:30.141612 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.141589 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5b64d\"" Nov 02 20:03:30.213541 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.213510 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939030d5-3b39-4d34-9b74-f0d5a464415a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.213723 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.213585 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjq46\" (UniqueName: \"kubernetes.io/projected/939030d5-3b39-4d34-9b74-f0d5a464415a-kube-api-access-mjq46\") pod \"payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.315193 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.315118 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939030d5-3b39-4d34-9b74-f0d5a464415a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.315193 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.315159 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjq46\" (UniqueName: \"kubernetes.io/projected/939030d5-3b39-4d34-9b74-f0d5a464415a-kube-api-access-mjq46\") pod \"payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.315947 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.315924 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939030d5-3b39-4d34-9b74-f0d5a464415a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.368040 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.368001 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjq46\" (UniqueName: \"kubernetes.io/projected/939030d5-3b39-4d34-9b74-f0d5a464415a-kube-api-access-mjq46\") pod \"payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.408950 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.408918 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:30.782209 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:30.782173 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm"] Nov 02 20:03:30.783091 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:30.783061 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939030d5_3b39_4d34_9b74_f0d5a464415a.slice/crio-e2244e003b55b3964a48545b258ed82b693326b94dbbde3ffe59e110afbf1abc WatchSource:0}: Error finding container e2244e003b55b3964a48545b258ed82b693326b94dbbde3ffe59e110afbf1abc: Status 404 returned error can't find the container with id e2244e003b55b3964a48545b258ed82b693326b94dbbde3ffe59e110afbf1abc Nov 02 20:03:31.602892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.602841 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" event={"ID":"939030d5-3b39-4d34-9b74-f0d5a464415a","Type":"ContainerStarted","Data":"e762434b20a0df531a3974e3c4207932a3ab4d359745b4d1737974b43544b103"} Nov 02 20:03:31.603398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.602906 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" event={"ID":"939030d5-3b39-4d34-9b74-f0d5a464415a","Type":"ContainerStarted","Data":"e2244e003b55b3964a48545b258ed82b693326b94dbbde3ffe59e110afbf1abc"} Nov 02 20:03:31.603398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.602930 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:31.709810 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.709741 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" podStartSLOduration=2.709720243 podStartE2EDuration="2.709720243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:31.708011088 +0000 UTC m=+2117.656666928" watchObservedRunningTime="2026-11-02 20:03:31.709720243 +0000 UTC m=+2117.658376085" Nov 02 20:03:31.776232 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.776191 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc"] Nov 02 20:03:31.781851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.781823 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.831059 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.831024 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:03:31.831235 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.831165 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:03:31.831235 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.831211 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:03:31.831388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.831266 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:03:31.833519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.833481 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc"] Nov 02 20:03:31.866443 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.866263 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.866837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.866808 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.867020 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.867005 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh7lx\" (UniqueName: \"kubernetes.io/projected/33bc9697-7933-4aab-bb13-89899bf4707a-kube-api-access-kh7lx\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.968817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.968768 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.969053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.968921 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.969053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.968955 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh7lx\" (UniqueName: \"kubernetes.io/projected/33bc9697-7933-4aab-bb13-89899bf4707a-kube-api-access-kh7lx\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.969996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.969955 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:31.971987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:31.971953 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:32.029339 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:32.027391 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh7lx\" (UniqueName: \"kubernetes.io/projected/33bc9697-7933-4aab-bb13-89899bf4707a-kube-api-access-kh7lx\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:32.107041 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:32.105764 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:32.754534 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:32.754504 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bc9697_7933_4aab_bb13_89899bf4707a.slice/crio-a322eab341c7ff110c9b983c0a62876dbad05c19963574e87810b77bf3d5e662 WatchSource:0}: Error finding container a322eab341c7ff110c9b983c0a62876dbad05c19963574e87810b77bf3d5e662: Status 404 returned error can't find the container with id a322eab341c7ff110c9b983c0a62876dbad05c19963574e87810b77bf3d5e662 Nov 02 20:03:32.755741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:32.755712 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc"] Nov 02 20:03:33.623526 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:33.623491 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" event={"ID":"33bc9697-7933-4aab-bb13-89899bf4707a","Type":"ContainerStarted","Data":"9d315bbad194a47868b3ab719a8ca50723a9b3a7ca2b246cdac99c05eb65f700"} Nov 02 20:03:33.623761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:33.623551 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" event={"ID":"33bc9697-7933-4aab-bb13-89899bf4707a","Type":"ContainerStarted","Data":"a322eab341c7ff110c9b983c0a62876dbad05c19963574e87810b77bf3d5e662"} Nov 02 20:03:33.733900 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:33.733827 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" podStartSLOduration=2.733807382 podStartE2EDuration="2.733807382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:33.72695028 +0000 UTC m=+2119.675606145" watchObservedRunningTime="2026-11-02 20:03:33.733807382 +0000 UTC m=+2119.682463221" Nov 02 20:03:34.633618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:34.633579 4966 generic.go:356] "Generic (PLEG): container finished" podID="33bc9697-7933-4aab-bb13-89899bf4707a" containerID="9d315bbad194a47868b3ab719a8ca50723a9b3a7ca2b246cdac99c05eb65f700" exitCode=0 Nov 02 20:03:34.634109 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:34.633646 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" event={"ID":"33bc9697-7933-4aab-bb13-89899bf4707a","Type":"ContainerDied","Data":"9d315bbad194a47868b3ab719a8ca50723a9b3a7ca2b246cdac99c05eb65f700"} Nov 02 20:03:35.257436 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.257392 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs"] Nov 02 20:03:35.301509 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.301474 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-4fd0d5-dd2qs"] Nov 02 20:03:35.852198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.852169 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:35.899639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.899557 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bc9697-7933-4aab-bb13-89899bf4707a-kube-api-access-kh7lx\" (UniqueName: \"kubernetes.io/projected/33bc9697-7933-4aab-bb13-89899bf4707a-kube-api-access-kh7lx\") pod \"33bc9697-7933-4aab-bb13-89899bf4707a\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " Nov 02 20:03:35.899639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.899662 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-openshift-ca\") pod \"33bc9697-7933-4aab-bb13-89899bf4707a\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " Nov 02 20:03:35.899959 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.899761 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-xks-ca\") pod \"33bc9697-7933-4aab-bb13-89899bf4707a\" (UID: \"33bc9697-7933-4aab-bb13-89899bf4707a\") " Nov 02 20:03:35.900086 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.900051 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-openshift-ca" pod "33bc9697-7933-4aab-bb13-89899bf4707a" (UID: "33bc9697-7933-4aab-bb13-89899bf4707a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:35.900445 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.900412 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:35.902141 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.902113 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bc9697-7933-4aab-bb13-89899bf4707a-kube-api-access-kh7lx" pod "33bc9697-7933-4aab-bb13-89899bf4707a" (UID: "33bc9697-7933-4aab-bb13-89899bf4707a"). InnerVolumeSpecName "kube-api-access-kh7lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:35.902529 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:35.902503 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-xks-ca" pod "33bc9697-7933-4aab-bb13-89899bf4707a" (UID: "33bc9697-7933-4aab-bb13-89899bf4707a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:36.002459 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:36.002413 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33bc9697-7933-4aab-bb13-89899bf4707a-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.002459 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:36.002456 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh7lx\" (UniqueName: \"kubernetes.io/projected/33bc9697-7933-4aab-bb13-89899bf4707a-kube-api-access-kh7lx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:36.186535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:36.186438 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c" path="/var/lib/kubelet/pods/0fb8a738-05b9-4350-a5f5-5be8ac3ffd0c/volumes" Nov 02 20:03:36.653676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:36.653649 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" Nov 02 20:03:36.653899 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:36.653646 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc" event={"ID":"33bc9697-7933-4aab-bb13-89899bf4707a","Type":"ContainerDied","Data":"a322eab341c7ff110c9b983c0a62876dbad05c19963574e87810b77bf3d5e662"} Nov 02 20:03:36.653899 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:36.653795 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a322eab341c7ff110c9b983c0a62876dbad05c19963574e87810b77bf3d5e662" Nov 02 20:03:40.891404 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.891360 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs"] Nov 02 20:03:40.892985 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.892954 4966 cpu_manager.go:414] "containerMap: removing container" podUID="33bc9697-7933-4aab-bb13-89899bf4707a" containerName="revoke-keys" Nov 02 20:03:40.893143 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.893129 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33bc9697-7933-4aab-bb13-89899bf4707a" containerName="revoke-keys" Nov 02 20:03:40.893472 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.893452 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="33bc9697-7933-4aab-bb13-89899bf4707a" containerName="revoke-keys" Nov 02 20:03:40.912335 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.905761 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:40.932343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.931108 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs"] Nov 02 20:03:40.971263 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.971233 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-838bc1\"" Nov 02 20:03:40.993952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.993911 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86cp\" (UniqueName: \"kubernetes.io/projected/e62b3cb3-a13a-4404-9c9a-f33a21110843-kube-api-access-m86cp\") pod \"payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:40.994097 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:40.993979 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62b3cb3-a13a-4404-9c9a-f33a21110843-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:41.030091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.030065 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-838bc1-dockercfg-z8wqk\"" Nov 02 20:03:41.095103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.095068 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86cp\" (UniqueName: \"kubernetes.io/projected/e62b3cb3-a13a-4404-9c9a-f33a21110843-kube-api-access-m86cp\") pod \"payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:41.095248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.095122 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62b3cb3-a13a-4404-9c9a-f33a21110843-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:41.095929 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.095904 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62b3cb3-a13a-4404-9c9a-f33a21110843-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:41.148066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.147994 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86cp\" (UniqueName: \"kubernetes.io/projected/e62b3cb3-a13a-4404-9c9a-f33a21110843-kube-api-access-m86cp\") pod \"payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:41.228711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.228664 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:41.610605 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:41.610561 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62b3cb3_a13a_4404_9c9a_f33a21110843.slice/crio-de4c00a261b7d27039316288ec7cc5e11198f9693893ad21033eb250f3fb9711 WatchSource:0}: Error finding container de4c00a261b7d27039316288ec7cc5e11198f9693893ad21033eb250f3fb9711: Status 404 returned error can't find the container with id de4c00a261b7d27039316288ec7cc5e11198f9693893ad21033eb250f3fb9711 Nov 02 20:03:41.610990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.610961 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs"] Nov 02 20:03:41.703253 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:41.703224 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" event={"ID":"e62b3cb3-a13a-4404-9c9a-f33a21110843","Type":"ContainerStarted","Data":"de4c00a261b7d27039316288ec7cc5e11198f9693893ad21033eb250f3fb9711"} Nov 02 20:03:42.617310 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:42.617268 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:03:42.711971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:42.711929 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" event={"ID":"e62b3cb3-a13a-4404-9c9a-f33a21110843","Type":"ContainerStarted","Data":"515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f"} Nov 02 20:03:42.712187 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:42.711993 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:42.835514 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:42.835436 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" podStartSLOduration=2.835415497 podStartE2EDuration="2.835415497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:42.828803879 +0000 UTC m=+2128.777459719" watchObservedRunningTime="2026-11-02 20:03:42.835415497 +0000 UTC m=+2128.784071338" Nov 02 20:03:43.815781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.815743 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf"] Nov 02 20:03:43.820038 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.820021 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:43.874982 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.874949 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf"] Nov 02 20:03:43.915627 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.915589 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:03:43.915807 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.915670 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:03:43.915807 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.915670 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:03:43.915807 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.915730 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:03:43.940594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.940562 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:43.940761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.940665 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:43.940761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:43.940689 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgndq\" (UniqueName: \"kubernetes.io/projected/37301b0f-5466-482e-a864-0bcde862b7bb-kube-api-access-fgndq\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.041980 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.041936 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.042190 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.042094 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.042190 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.042126 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgndq\" (UniqueName: \"kubernetes.io/projected/37301b0f-5466-482e-a864-0bcde862b7bb-kube-api-access-fgndq\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.042860 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.042837 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.046139 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.046109 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.095107 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.095038 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgndq\" (UniqueName: \"kubernetes.io/projected/37301b0f-5466-482e-a864-0bcde862b7bb-kube-api-access-fgndq\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.133974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.133932 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:44.547626 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.538653 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf"] Nov 02 20:03:44.563866 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:44.561037 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37301b0f_5466_482e_a864_0bcde862b7bb.slice/crio-eb25e37788db3dd6174eaf12c354dbcbf62bd744c844759ceb14c006fa7469b7 WatchSource:0}: Error finding container eb25e37788db3dd6174eaf12c354dbcbf62bd744c844759ceb14c006fa7469b7: Status 404 returned error can't find the container with id eb25e37788db3dd6174eaf12c354dbcbf62bd744c844759ceb14c006fa7469b7 Nov 02 20:03:44.728266 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:44.728228 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" event={"ID":"37301b0f-5466-482e-a864-0bcde862b7bb","Type":"ContainerStarted","Data":"eb25e37788db3dd6174eaf12c354dbcbf62bd744c844759ceb14c006fa7469b7"} Nov 02 20:03:44.755083 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:03:44.755047 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:03:45.311923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:45.311884 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf"] Nov 02 20:03:45.432917 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:45.432881 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s"] Nov 02 20:03:45.738536 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:45.738491 4966 generic.go:356] "Generic (PLEG): container finished" podID="37301b0f-5466-482e-a864-0bcde862b7bb" containerID="62324e96e336196597a6ccd33ba7f1526db4161bc0b49aca3932e647b7ef0029" exitCode=0 Nov 02 20:03:45.738780 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:45.738757 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" event={"ID":"37301b0f-5466-482e-a864-0bcde862b7bb","Type":"ContainerDied","Data":"62324e96e336196597a6ccd33ba7f1526db4161bc0b49aca3932e647b7ef0029"} Nov 02 20:03:46.196725 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:46.196693 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt"] Nov 02 20:03:46.201508 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:46.201481 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0cc24-t9klt"] Nov 02 20:03:47.041516 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.041490 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:47.146832 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.145695 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-xks-ca\") pod \"37301b0f-5466-482e-a864-0bcde862b7bb\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " Nov 02 20:03:47.146832 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.145951 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-openshift-ca\") pod \"37301b0f-5466-482e-a864-0bcde862b7bb\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " Nov 02 20:03:47.146832 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.146011 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37301b0f-5466-482e-a864-0bcde862b7bb-kube-api-access-fgndq\" (UniqueName: \"kubernetes.io/projected/37301b0f-5466-482e-a864-0bcde862b7bb-kube-api-access-fgndq\") pod \"37301b0f-5466-482e-a864-0bcde862b7bb\" (UID: \"37301b0f-5466-482e-a864-0bcde862b7bb\") " Nov 02 20:03:47.149734 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.149695 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37301b0f-5466-482e-a864-0bcde862b7bb-kube-api-access-fgndq" pod "37301b0f-5466-482e-a864-0bcde862b7bb" (UID: "37301b0f-5466-482e-a864-0bcde862b7bb"). InnerVolumeSpecName "kube-api-access-fgndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:47.150241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.150216 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-openshift-ca" pod "37301b0f-5466-482e-a864-0bcde862b7bb" (UID: "37301b0f-5466-482e-a864-0bcde862b7bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:47.167452 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.158585 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-xks-ca" pod "37301b0f-5466-482e-a864-0bcde862b7bb" (UID: "37301b0f-5466-482e-a864-0bcde862b7bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:47.265380 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.263108 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:47.265380 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.263153 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37301b0f-5466-482e-a864-0bcde862b7bb-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:47.265380 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.263171 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgndq\" (UniqueName: \"kubernetes.io/projected/37301b0f-5466-482e-a864-0bcde862b7bb-kube-api-access-fgndq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:47.778636 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.778480 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" event={"ID":"37301b0f-5466-482e-a864-0bcde862b7bb","Type":"ContainerDied","Data":"eb25e37788db3dd6174eaf12c354dbcbf62bd744c844759ceb14c006fa7469b7"} Nov 02 20:03:47.778636 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.778601 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb25e37788db3dd6174eaf12c354dbcbf62bd744c844759ceb14c006fa7469b7" Nov 02 20:03:47.779004 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:47.778979 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf" Nov 02 20:03:48.186008 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:48.185914 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27652d1d-d665-4e95-b9b5-b231cb1a5202" path="/var/lib/kubelet/pods/27652d1d-d665-4e95-b9b5-b231cb1a5202/volumes" Nov 02 20:03:50.312562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.312500 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" podUID="af6b0606-6b1c-4b6c-903a-1f8532f877a7" containerName="payload-processing" containerID="cri-o://18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230" gracePeriod=25 Nov 02 20:03:50.433844 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.433726 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" podUID="2b1d7daa-ab6e-4613-ab73-5497cc565854" containerName="payload-pre-processing" containerID="cri-o://819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517" gracePeriod=25 Nov 02 20:03:50.621571 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.621447 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 20:03:50.626756 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.626726 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6b0606-6b1c-4b6c-903a-1f8532f877a7-kube-api-access-lqljx\" (UniqueName: \"kubernetes.io/projected/af6b0606-6b1c-4b6c-903a-1f8532f877a7-kube-api-access-lqljx\") pod \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " Nov 02 20:03:50.627008 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.626989 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6b0606-6b1c-4b6c-903a-1f8532f877a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b0606-6b1c-4b6c-903a-1f8532f877a7-plugins-config-volume\") pod \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\" (UID: \"af6b0606-6b1c-4b6c-903a-1f8532f877a7\") " Nov 02 20:03:50.627590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.627560 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6b0606-6b1c-4b6c-903a-1f8532f877a7-plugins-config-volume" pod "af6b0606-6b1c-4b6c-903a-1f8532f877a7" (UID: "af6b0606-6b1c-4b6c-903a-1f8532f877a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:50.628936 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.628893 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6b0606-6b1c-4b6c-903a-1f8532f877a7-kube-api-access-lqljx" pod "af6b0606-6b1c-4b6c-903a-1f8532f877a7" (UID: "af6b0606-6b1c-4b6c-903a-1f8532f877a7"). InnerVolumeSpecName "kube-api-access-lqljx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:50.727240 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.727192 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 20:03:50.728890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.728862 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqljx\" (UniqueName: \"kubernetes.io/projected/af6b0606-6b1c-4b6c-903a-1f8532f877a7-kube-api-access-lqljx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:50.729067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.729053 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6b0606-6b1c-4b6c-903a-1f8532f877a7-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:50.810507 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.810469 4966 generic.go:356] "Generic (PLEG): container finished" podID="af6b0606-6b1c-4b6c-903a-1f8532f877a7" containerID="18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230" exitCode=0 Nov 02 20:03:50.810689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.810521 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" Nov 02 20:03:50.810689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.810555 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" event={"ID":"af6b0606-6b1c-4b6c-903a-1f8532f877a7","Type":"ContainerDied","Data":"18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230"} Nov 02 20:03:50.810689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.810613 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf" event={"ID":"af6b0606-6b1c-4b6c-903a-1f8532f877a7","Type":"ContainerDied","Data":"164cff35b2cddf2205c55e15830aeb1ca86b4d7694fff81aae9dfcaddf1939ba"} Nov 02 20:03:50.810689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.810635 4966 scope.go:122] "RemoveContainer" containerID="18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230" Nov 02 20:03:50.812237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.812214 4966 generic.go:356] "Generic (PLEG): container finished" podID="2b1d7daa-ab6e-4613-ab73-5497cc565854" containerID="819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517" exitCode=0 Nov 02 20:03:50.812377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.812246 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" event={"ID":"2b1d7daa-ab6e-4613-ab73-5497cc565854","Type":"ContainerDied","Data":"819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517"} Nov 02 20:03:50.812377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.812276 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" Nov 02 20:03:50.812377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.812333 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s" event={"ID":"2b1d7daa-ab6e-4613-ab73-5497cc565854","Type":"ContainerDied","Data":"7933de30d8f799a12c2a9d2e9271273a36c473ffafe2162e61deecc52be954e1"} Nov 02 20:03:50.822318 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.822285 4966 scope.go:122] "RemoveContainer" containerID="18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230" Nov 02 20:03:50.822607 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:03:50.822586 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230\": container with ID starting with 18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230 not found: ID does not exist" containerID="18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230" Nov 02 20:03:50.822700 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.822617 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230"} err="failed to get container status \"18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230\": rpc error: code = NotFound desc = could not find container \"18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230\": container with ID starting with 18b7fa7c5746062a72e6dcf4e765ddfeaaf9e86eac6e711eb1c142f4280e5230 not found: ID does not exist" Nov 02 20:03:50.822700 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.822634 4966 scope.go:122] "RemoveContainer" containerID="819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517" Nov 02 20:03:50.830091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.830064 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1d7daa-ab6e-4613-ab73-5497cc565854-kube-api-access-qt8cj\" (UniqueName: \"kubernetes.io/projected/2b1d7daa-ab6e-4613-ab73-5497cc565854-kube-api-access-qt8cj\") pod \"2b1d7daa-ab6e-4613-ab73-5497cc565854\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " Nov 02 20:03:50.830329 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.830313 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1d7daa-ab6e-4613-ab73-5497cc565854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1d7daa-ab6e-4613-ab73-5497cc565854-plugins-config-volume\") pod \"2b1d7daa-ab6e-4613-ab73-5497cc565854\" (UID: \"2b1d7daa-ab6e-4613-ab73-5497cc565854\") " Nov 02 20:03:50.830684 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.830662 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1d7daa-ab6e-4613-ab73-5497cc565854-plugins-config-volume" pod "2b1d7daa-ab6e-4613-ab73-5497cc565854" (UID: "2b1d7daa-ab6e-4613-ab73-5497cc565854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:50.830872 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.830855 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1d7daa-ab6e-4613-ab73-5497cc565854-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:50.832740 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.832711 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1d7daa-ab6e-4613-ab73-5497cc565854-kube-api-access-qt8cj" pod "2b1d7daa-ab6e-4613-ab73-5497cc565854" (UID: "2b1d7daa-ab6e-4613-ab73-5497cc565854"). InnerVolumeSpecName "kube-api-access-qt8cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:50.843574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.843491 4966 scope.go:122] "RemoveContainer" containerID="819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517" Nov 02 20:03:50.845671 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:03:50.845204 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517\": container with ID starting with 819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517 not found: ID does not exist" containerID="819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517" Nov 02 20:03:50.845671 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.845277 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517"} err="failed to get container status \"819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517\": rpc error: code = NotFound desc = could not find container \"819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517\": container with ID starting with 819ccbf71d0fd1aceef7a43f17dfff8ef26f3920673323ec0f25a2e719214517 not found: ID does not exist" Nov 02 20:03:50.931830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.931800 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt8cj\" (UniqueName: \"kubernetes.io/projected/2b1d7daa-ab6e-4613-ab73-5497cc565854-kube-api-access-qt8cj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:50.937237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.937210 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf"] Nov 02 20:03:50.942342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:50.942310 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ffdc4d-d8c44465f-jf7tf"] Nov 02 20:03:51.226872 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:51.226828 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s"] Nov 02 20:03:51.231238 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:51.231201 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ffdc4d-d6c958bff-v6k2s"] Nov 02 20:03:52.187659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:52.187628 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1d7daa-ab6e-4613-ab73-5497cc565854" path="/var/lib/kubelet/pods/2b1d7daa-ab6e-4613-ab73-5497cc565854/volumes" Nov 02 20:03:52.188132 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:52.188117 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6b0606-6b1c-4b6c-903a-1f8532f877a7" path="/var/lib/kubelet/pods/af6b0606-6b1c-4b6c-903a-1f8532f877a7/volumes" Nov 02 20:03:53.503405 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.503360 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb"] Nov 02 20:03:53.504199 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504172 4966 cpu_manager.go:414] "containerMap: removing container" podUID="2b1d7daa-ab6e-4613-ab73-5497cc565854" containerName="payload-pre-processing" Nov 02 20:03:53.504326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504201 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1d7daa-ab6e-4613-ab73-5497cc565854" containerName="payload-pre-processing" Nov 02 20:03:53.504326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504243 4966 cpu_manager.go:414] "containerMap: removing container" podUID="af6b0606-6b1c-4b6c-903a-1f8532f877a7" containerName="payload-processing" Nov 02 20:03:53.504326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504253 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6b0606-6b1c-4b6c-903a-1f8532f877a7" containerName="payload-processing" Nov 02 20:03:53.504326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504271 4966 cpu_manager.go:414] "containerMap: removing container" podUID="37301b0f-5466-482e-a864-0bcde862b7bb" containerName="revoke-keys" Nov 02 20:03:53.504326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504278 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37301b0f-5466-482e-a864-0bcde862b7bb" containerName="revoke-keys" Nov 02 20:03:53.504598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504497 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6b0606-6b1c-4b6c-903a-1f8532f877a7" containerName="payload-processing" Nov 02 20:03:53.504598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504515 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="37301b0f-5466-482e-a864-0bcde862b7bb" containerName="revoke-keys" Nov 02 20:03:53.504598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.504536 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1d7daa-ab6e-4613-ab73-5497cc565854" containerName="payload-pre-processing" Nov 02 20:03:53.509890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.509868 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.561821 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.561789 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb"] Nov 02 20:03:53.572090 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.572060 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfljq\" (UniqueName: \"kubernetes.io/projected/1006eb85-72d7-4c65-b2b9-e06d14591848-kube-api-access-wfljq\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.572342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.572289 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.572441 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.572339 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.602604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.602564 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:03:53.602763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.602605 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:03:53.605208 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.605190 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:03:53.605272 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.605261 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:03:53.673786 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.673747 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.673974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.673792 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.673974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.673849 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfljq\" (UniqueName: \"kubernetes.io/projected/1006eb85-72d7-4c65-b2b9-e06d14591848-kube-api-access-wfljq\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.674576 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.674549 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.676401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.676374 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.723233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.723208 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:03:53.726851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.726824 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfljq\" (UniqueName: \"kubernetes.io/projected/1006eb85-72d7-4c65-b2b9-e06d14591848-kube-api-access-wfljq\") pod \"maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:53.824601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:53.824526 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:54.214751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:54.214720 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb"] Nov 02 20:03:54.215222 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:03:54.215191 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1006eb85_72d7_4c65_b2b9_e06d14591848.slice/crio-1e0935dafc6b656849a27cfd8bf780d1670838e3de223414a4ab9a4317078b04 WatchSource:0}: Error finding container 1e0935dafc6b656849a27cfd8bf780d1670838e3de223414a4ab9a4317078b04: Status 404 returned error can't find the container with id 1e0935dafc6b656849a27cfd8bf780d1670838e3de223414a4ab9a4317078b04 Nov 02 20:03:54.893393 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:54.893275 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" event={"ID":"1006eb85-72d7-4c65-b2b9-e06d14591848","Type":"ContainerStarted","Data":"d79bcc50e4880021cb3911de794e74eadc91529d9685ca538180e4b4d11b5e54"} Nov 02 20:03:54.893393 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:54.893368 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" event={"ID":"1006eb85-72d7-4c65-b2b9-e06d14591848","Type":"ContainerStarted","Data":"1e0935dafc6b656849a27cfd8bf780d1670838e3de223414a4ab9a4317078b04"} Nov 02 20:03:54.998330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:54.998239 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" podStartSLOduration=1.9982247960000001 podStartE2EDuration="1.998224796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:54.996904462 +0000 UTC m=+2140.945560301" watchObservedRunningTime="2026-11-02 20:03:54.998224796 +0000 UTC m=+2140.946880634" Nov 02 20:03:55.901834 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:55.901798 4966 generic.go:356] "Generic (PLEG): container finished" podID="1006eb85-72d7-4c65-b2b9-e06d14591848" containerID="d79bcc50e4880021cb3911de794e74eadc91529d9685ca538180e4b4d11b5e54" exitCode=0 Nov 02 20:03:55.902271 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:55.901883 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" event={"ID":"1006eb85-72d7-4c65-b2b9-e06d14591848","Type":"ContainerDied","Data":"d79bcc50e4880021cb3911de794e74eadc91529d9685ca538180e4b4d11b5e54"} Nov 02 20:03:57.079415 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.079384 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:57.130017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.129986 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-xks-ca\") pod \"1006eb85-72d7-4c65-b2b9-e06d14591848\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " Nov 02 20:03:57.130184 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.130131 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-openshift-ca\") pod \"1006eb85-72d7-4c65-b2b9-e06d14591848\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " Nov 02 20:03:57.130291 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.130273 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1006eb85-72d7-4c65-b2b9-e06d14591848-kube-api-access-wfljq\" (UniqueName: \"kubernetes.io/projected/1006eb85-72d7-4c65-b2b9-e06d14591848-kube-api-access-wfljq\") pod \"1006eb85-72d7-4c65-b2b9-e06d14591848\" (UID: \"1006eb85-72d7-4c65-b2b9-e06d14591848\") " Nov 02 20:03:57.130538 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.130514 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-openshift-ca" pod "1006eb85-72d7-4c65-b2b9-e06d14591848" (UID: "1006eb85-72d7-4c65-b2b9-e06d14591848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:57.130820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.130802 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:57.132155 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.132124 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-xks-ca" pod "1006eb85-72d7-4c65-b2b9-e06d14591848" (UID: "1006eb85-72d7-4c65-b2b9-e06d14591848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:57.132268 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.132223 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1006eb85-72d7-4c65-b2b9-e06d14591848-kube-api-access-wfljq" pod "1006eb85-72d7-4c65-b2b9-e06d14591848" (UID: "1006eb85-72d7-4c65-b2b9-e06d14591848"). InnerVolumeSpecName "kube-api-access-wfljq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:57.231971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.231926 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfljq\" (UniqueName: \"kubernetes.io/projected/1006eb85-72d7-4c65-b2b9-e06d14591848-kube-api-access-wfljq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:57.231971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.231960 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1006eb85-72d7-4c65-b2b9-e06d14591848-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:57.920771 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.920742 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" Nov 02 20:03:57.920962 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.920770 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb" event={"ID":"1006eb85-72d7-4c65-b2b9-e06d14591848","Type":"ContainerDied","Data":"1e0935dafc6b656849a27cfd8bf780d1670838e3de223414a4ab9a4317078b04"} Nov 02 20:03:57.920962 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:03:57.920826 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e0935dafc6b656849a27cfd8bf780d1670838e3de223414a4ab9a4317078b04" Nov 02 20:04:04.669466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:04.669383 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft"] Nov 02 20:04:04.787957 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:04.787920 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz"] Nov 02 20:04:09.669736 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:09.669679 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" podUID="c435f75f-ad8b-40fb-b461-fbf070cada9d" containerName="payload-processing" containerID="cri-o://f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf" gracePeriod=25 Nov 02 20:04:09.789468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:09.789418 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" podUID="44782bb2-91ba-4c84-bae8-529bb304390a" containerName="payload-pre-processing" containerID="cri-o://dfe22961548e3472a860885c176fdfda2807587598c26831db57c3cf2a6c135e" gracePeriod=25 Nov 02 20:04:09.975371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:09.975200 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 20:04:10.032147 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.032099 4966 generic.go:356] "Generic (PLEG): container finished" podID="44782bb2-91ba-4c84-bae8-529bb304390a" containerID="dfe22961548e3472a860885c176fdfda2807587598c26831db57c3cf2a6c135e" exitCode=0 Nov 02 20:04:10.033709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.032251 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" event={"ID":"44782bb2-91ba-4c84-bae8-529bb304390a","Type":"ContainerDied","Data":"dfe22961548e3472a860885c176fdfda2807587598c26831db57c3cf2a6c135e"} Nov 02 20:04:10.035607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.035579 4966 generic.go:356] "Generic (PLEG): container finished" podID="c435f75f-ad8b-40fb-b461-fbf070cada9d" containerID="f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf" exitCode=0 Nov 02 20:04:10.035761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.035635 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" Nov 02 20:04:10.035761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.035709 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" event={"ID":"c435f75f-ad8b-40fb-b461-fbf070cada9d","Type":"ContainerDied","Data":"f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf"} Nov 02 20:04:10.035891 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.035761 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft" event={"ID":"c435f75f-ad8b-40fb-b461-fbf070cada9d","Type":"ContainerDied","Data":"1fad4d1ca0f8132a9d5fd2c5d73ded8378a5834d2f3c3733b66c743d6bc461bb"} Nov 02 20:04:10.035891 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.035794 4966 scope.go:122] "RemoveContainer" containerID="f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf" Nov 02 20:04:10.101636 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.101595 4966 scope.go:122] "RemoveContainer" containerID="f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf" Nov 02 20:04:10.103639 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:04:10.103615 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf\": container with ID starting with f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf not found: ID does not exist" containerID="f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf" Nov 02 20:04:10.103740 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.103648 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf"} err="failed to get container status \"f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf\": rpc error: code = NotFound desc = could not find container \"f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf\": container with ID starting with f1156fe1dc2509233f2ed7532570277c8e87794ff6142dafd593af340b61a8bf not found: ID does not exist" Nov 02 20:04:10.120558 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.120538 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c435f75f-ad8b-40fb-b461-fbf070cada9d-kube-api-access-jxxpg\" (UniqueName: \"kubernetes.io/projected/c435f75f-ad8b-40fb-b461-fbf070cada9d-kube-api-access-jxxpg\") pod \"c435f75f-ad8b-40fb-b461-fbf070cada9d\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " Nov 02 20:04:10.121019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.120996 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c435f75f-ad8b-40fb-b461-fbf070cada9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c435f75f-ad8b-40fb-b461-fbf070cada9d-plugins-config-volume\") pod \"c435f75f-ad8b-40fb-b461-fbf070cada9d\" (UID: \"c435f75f-ad8b-40fb-b461-fbf070cada9d\") " Nov 02 20:04:10.121385 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.121362 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c435f75f-ad8b-40fb-b461-fbf070cada9d-plugins-config-volume" pod "c435f75f-ad8b-40fb-b461-fbf070cada9d" (UID: "c435f75f-ad8b-40fb-b461-fbf070cada9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:10.121840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.121817 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c435f75f-ad8b-40fb-b461-fbf070cada9d-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:10.123976 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.123955 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c435f75f-ad8b-40fb-b461-fbf070cada9d-kube-api-access-jxxpg" pod "c435f75f-ad8b-40fb-b461-fbf070cada9d" (UID: "c435f75f-ad8b-40fb-b461-fbf070cada9d"). InnerVolumeSpecName "kube-api-access-jxxpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:10.133949 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.133927 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 20:04:10.222837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.222802 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44782bb2-91ba-4c84-bae8-529bb304390a-kube-api-access-6v7gk\" (UniqueName: \"kubernetes.io/projected/44782bb2-91ba-4c84-bae8-529bb304390a-kube-api-access-6v7gk\") pod \"44782bb2-91ba-4c84-bae8-529bb304390a\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " Nov 02 20:04:10.223031 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.223004 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44782bb2-91ba-4c84-bae8-529bb304390a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44782bb2-91ba-4c84-bae8-529bb304390a-plugins-config-volume\") pod \"44782bb2-91ba-4c84-bae8-529bb304390a\" (UID: \"44782bb2-91ba-4c84-bae8-529bb304390a\") " Nov 02 20:04:10.223313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.223276 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44782bb2-91ba-4c84-bae8-529bb304390a-plugins-config-volume" pod "44782bb2-91ba-4c84-bae8-529bb304390a" (UID: "44782bb2-91ba-4c84-bae8-529bb304390a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:10.223545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.223527 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxxpg\" (UniqueName: \"kubernetes.io/projected/c435f75f-ad8b-40fb-b461-fbf070cada9d-kube-api-access-jxxpg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:10.223593 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.223547 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44782bb2-91ba-4c84-bae8-529bb304390a-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:10.224939 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.224918 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44782bb2-91ba-4c84-bae8-529bb304390a-kube-api-access-6v7gk" pod "44782bb2-91ba-4c84-bae8-529bb304390a" (UID: "44782bb2-91ba-4c84-bae8-529bb304390a"). InnerVolumeSpecName "kube-api-access-6v7gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:10.325004 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.324970 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v7gk\" (UniqueName: \"kubernetes.io/projected/44782bb2-91ba-4c84-bae8-529bb304390a-kube-api-access-6v7gk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:10.446462 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.446421 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft"] Nov 02 20:04:10.452676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:10.452642 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b0a12-67fd94f9d7-w4wft"] Nov 02 20:04:11.048748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:11.048717 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" Nov 02 20:04:11.048748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:11.048738 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz" event={"ID":"44782bb2-91ba-4c84-bae8-529bb304390a","Type":"ContainerDied","Data":"00c861aa654802e15a56d98379c775ba654fc13fa626748bb9c726d79875c7b2"} Nov 02 20:04:11.049352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:11.048804 4966 scope.go:122] "RemoveContainer" containerID="dfe22961548e3472a860885c176fdfda2807587598c26831db57c3cf2a6c135e" Nov 02 20:04:11.202617 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:11.202577 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz"] Nov 02 20:04:11.249100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:11.249060 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b0a12-868fbb74f9-lghhz"] Nov 02 20:04:12.186031 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:12.185991 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44782bb2-91ba-4c84-bae8-529bb304390a" path="/var/lib/kubelet/pods/44782bb2-91ba-4c84-bae8-529bb304390a/volumes" Nov 02 20:04:12.186598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:12.186577 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c435f75f-ad8b-40fb-b461-fbf070cada9d" path="/var/lib/kubelet/pods/c435f75f-ad8b-40fb-b461-fbf070cada9d/volumes" Nov 02 20:04:14.753421 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:04:14.753385 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:04:18.408996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:18.408962 4966 scope.go:122] "RemoveContainer" containerID="ab1198d675b62c9800749d5afdd2782b6c39ad3127c1c2d1d15d4bc24130f686" Nov 02 20:04:41.257848 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:41.257800 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26"] Nov 02 20:04:41.303367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:41.303331 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b7eed-z7v26"] Nov 02 20:04:42.184955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:42.184918 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9f8715-6b8f-4350-aa36-8859e468a1c1" path="/var/lib/kubelet/pods/1a9f8715-6b8f-4350-aa36-8859e468a1c1/volumes" Nov 02 20:04:44.756138 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:04:44.756101 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:04:48.605886 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.605845 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz"] Nov 02 20:04:48.606615 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606591 4966 cpu_manager.go:414] "containerMap: removing container" podUID="44782bb2-91ba-4c84-bae8-529bb304390a" containerName="payload-pre-processing" Nov 02 20:04:48.606615 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606615 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44782bb2-91ba-4c84-bae8-529bb304390a" containerName="payload-pre-processing" Nov 02 20:04:48.606774 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606634 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1006eb85-72d7-4c65-b2b9-e06d14591848" containerName="revoke-keys" Nov 02 20:04:48.606774 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606640 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1006eb85-72d7-4c65-b2b9-e06d14591848" containerName="revoke-keys" Nov 02 20:04:48.606774 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606679 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c435f75f-ad8b-40fb-b461-fbf070cada9d" containerName="payload-processing" Nov 02 20:04:48.606774 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606685 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c435f75f-ad8b-40fb-b461-fbf070cada9d" containerName="payload-processing" Nov 02 20:04:48.606909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606807 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1006eb85-72d7-4c65-b2b9-e06d14591848" containerName="revoke-keys" Nov 02 20:04:48.606909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606818 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c435f75f-ad8b-40fb-b461-fbf070cada9d" containerName="payload-processing" Nov 02 20:04:48.606909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.606831 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="44782bb2-91ba-4c84-bae8-529bb304390a" containerName="payload-pre-processing" Nov 02 20:04:48.610731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.610714 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.662408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.662374 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz"] Nov 02 20:04:48.703083 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.703053 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:04:48.703385 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.703089 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:04:48.704776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.704742 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:04:48.704776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.704751 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:04:48.749809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.749766 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.749809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.749812 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxxns\" (UniqueName: \"kubernetes.io/projected/4921acd7-9e9e-450c-8723-a13bac16040e-kube-api-access-jxxns\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.750127 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.750104 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.851182 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.851143 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.851424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.851241 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.851424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.851271 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxxns\" (UniqueName: \"kubernetes.io/projected/4921acd7-9e9e-450c-8723-a13bac16040e-kube-api-access-jxxns\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.851888 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.851860 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.853945 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.853924 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.904381 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.904270 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxxns\" (UniqueName: \"kubernetes.io/projected/4921acd7-9e9e-450c-8723-a13bac16040e-kube-api-access-jxxns\") pod \"maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:48.924795 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:48.924766 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:49.294946 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:49.294909 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz"] Nov 02 20:04:49.295468 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:04:49.295435 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4921acd7_9e9e_450c_8723_a13bac16040e.slice/crio-3a28d4dc83ff8d88ace0d0bcc9fade609464482a6ea2436712fd620af0e0db19 WatchSource:0}: Error finding container 3a28d4dc83ff8d88ace0d0bcc9fade609464482a6ea2436712fd620af0e0db19: Status 404 returned error can't find the container with id 3a28d4dc83ff8d88ace0d0bcc9fade609464482a6ea2436712fd620af0e0db19 Nov 02 20:04:49.398148 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:49.398110 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" event={"ID":"4921acd7-9e9e-450c-8723-a13bac16040e","Type":"ContainerStarted","Data":"3a28d4dc83ff8d88ace0d0bcc9fade609464482a6ea2436712fd620af0e0db19"} Nov 02 20:04:50.409204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:50.409099 4966 generic.go:356] "Generic (PLEG): container finished" podID="4921acd7-9e9e-450c-8723-a13bac16040e" containerID="85bfa4ac0bb1f9633715b5bae983df57fc01b7ed374f4bf4f11999ba44180f3a" exitCode=0 Nov 02 20:04:50.409689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:50.409194 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" event={"ID":"4921acd7-9e9e-450c-8723-a13bac16040e","Type":"ContainerDied","Data":"85bfa4ac0bb1f9633715b5bae983df57fc01b7ed374f4bf4f11999ba44180f3a"} Nov 02 20:04:51.599065 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.599041 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:51.688198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.688106 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-openshift-ca\") pod \"4921acd7-9e9e-450c-8723-a13bac16040e\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " Nov 02 20:04:51.688420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.688235 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4921acd7-9e9e-450c-8723-a13bac16040e-kube-api-access-jxxns\" (UniqueName: \"kubernetes.io/projected/4921acd7-9e9e-450c-8723-a13bac16040e-kube-api-access-jxxns\") pod \"4921acd7-9e9e-450c-8723-a13bac16040e\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " Nov 02 20:04:51.688420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.688397 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-xks-ca\") pod \"4921acd7-9e9e-450c-8723-a13bac16040e\" (UID: \"4921acd7-9e9e-450c-8723-a13bac16040e\") " Nov 02 20:04:51.688597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.688563 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-openshift-ca" pod "4921acd7-9e9e-450c-8723-a13bac16040e" (UID: "4921acd7-9e9e-450c-8723-a13bac16040e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:51.689079 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.689052 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:51.690653 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.690617 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4921acd7-9e9e-450c-8723-a13bac16040e-kube-api-access-jxxns" pod "4921acd7-9e9e-450c-8723-a13bac16040e" (UID: "4921acd7-9e9e-450c-8723-a13bac16040e"). InnerVolumeSpecName "kube-api-access-jxxns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:51.690768 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.690675 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-xks-ca" pod "4921acd7-9e9e-450c-8723-a13bac16040e" (UID: "4921acd7-9e9e-450c-8723-a13bac16040e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:51.789915 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.789868 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxxns\" (UniqueName: \"kubernetes.io/projected/4921acd7-9e9e-450c-8723-a13bac16040e-kube-api-access-jxxns\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:51.789915 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:51.789909 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4921acd7-9e9e-450c-8723-a13bac16040e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:52.428950 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:52.428917 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" Nov 02 20:04:52.429118 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:52.428920 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz" event={"ID":"4921acd7-9e9e-450c-8723-a13bac16040e","Type":"ContainerDied","Data":"3a28d4dc83ff8d88ace0d0bcc9fade609464482a6ea2436712fd620af0e0db19"} Nov 02 20:04:52.429118 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:04:52.429051 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a28d4dc83ff8d88ace0d0bcc9fade609464482a6ea2436712fd620af0e0db19" Nov 02 20:05:01.593004 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:01.592924 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc"] Nov 02 20:05:01.644905 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:01.644847 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ffdc4d-j6xsc"] Nov 02 20:05:02.186908 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:02.186874 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33bc9697-7933-4aab-bb13-89899bf4707a" path="/var/lib/kubelet/pods/33bc9697-7933-4aab-bb13-89899bf4707a/volumes" Nov 02 20:05:05.173341 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:05.173289 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl"] Nov 02 20:05:05.293159 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:05.293116 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l"] Nov 02 20:05:10.173802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.173745 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" podUID="aa48ba08-be58-4f1c-8834-99b9c46b999f" containerName="payload-processing" containerID="cri-o://a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76" gracePeriod=25 Nov 02 20:05:10.293904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.293812 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" podUID="15a0ed26-57d3-46d4-8535-82adda030709" containerName="payload-pre-processing" containerID="cri-o://535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0" gracePeriod=25 Nov 02 20:05:10.509319 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.508568 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 20:05:10.605145 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.605105 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa48ba08-be58-4f1c-8834-99b9c46b999f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa48ba08-be58-4f1c-8834-99b9c46b999f-plugins-config-volume\") pod \"aa48ba08-be58-4f1c-8834-99b9c46b999f\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " Nov 02 20:05:10.605389 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.605196 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa48ba08-be58-4f1c-8834-99b9c46b999f-kube-api-access-8ccrj\" (UniqueName: \"kubernetes.io/projected/aa48ba08-be58-4f1c-8834-99b9c46b999f-kube-api-access-8ccrj\") pod \"aa48ba08-be58-4f1c-8834-99b9c46b999f\" (UID: \"aa48ba08-be58-4f1c-8834-99b9c46b999f\") " Nov 02 20:05:10.605650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.605617 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa48ba08-be58-4f1c-8834-99b9c46b999f-plugins-config-volume" pod "aa48ba08-be58-4f1c-8834-99b9c46b999f" (UID: "aa48ba08-be58-4f1c-8834-99b9c46b999f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:10.605870 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.605839 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa48ba08-be58-4f1c-8834-99b9c46b999f-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:10.608479 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.608444 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa48ba08-be58-4f1c-8834-99b9c46b999f-kube-api-access-8ccrj" pod "aa48ba08-be58-4f1c-8834-99b9c46b999f" (UID: "aa48ba08-be58-4f1c-8834-99b9c46b999f"). InnerVolumeSpecName "kube-api-access-8ccrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:10.648445 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.648410 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 20:05:10.707166 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.707132 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a0ed26-57d3-46d4-8535-82adda030709-kube-api-access-9kgpv\" (UniqueName: \"kubernetes.io/projected/15a0ed26-57d3-46d4-8535-82adda030709-kube-api-access-9kgpv\") pod \"15a0ed26-57d3-46d4-8535-82adda030709\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " Nov 02 20:05:10.707166 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.707179 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a0ed26-57d3-46d4-8535-82adda030709-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a0ed26-57d3-46d4-8535-82adda030709-plugins-config-volume\") pod \"15a0ed26-57d3-46d4-8535-82adda030709\" (UID: \"15a0ed26-57d3-46d4-8535-82adda030709\") " Nov 02 20:05:10.707867 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.707837 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ccrj\" (UniqueName: \"kubernetes.io/projected/aa48ba08-be58-4f1c-8834-99b9c46b999f-kube-api-access-8ccrj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:10.708791 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.708760 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a0ed26-57d3-46d4-8535-82adda030709-plugins-config-volume" pod "15a0ed26-57d3-46d4-8535-82adda030709" (UID: "15a0ed26-57d3-46d4-8535-82adda030709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:10.709630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.709611 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a0ed26-57d3-46d4-8535-82adda030709-kube-api-access-9kgpv" pod "15a0ed26-57d3-46d4-8535-82adda030709" (UID: "15a0ed26-57d3-46d4-8535-82adda030709"). InnerVolumeSpecName "kube-api-access-9kgpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:10.809191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.809102 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a0ed26-57d3-46d4-8535-82adda030709-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:10.809191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.809137 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgpv\" (UniqueName: \"kubernetes.io/projected/15a0ed26-57d3-46d4-8535-82adda030709-kube-api-access-9kgpv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:10.834527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.834491 4966 generic.go:356] "Generic (PLEG): container finished" podID="15a0ed26-57d3-46d4-8535-82adda030709" containerID="535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0" exitCode=0 Nov 02 20:05:10.834728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.834556 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" Nov 02 20:05:10.834728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.834580 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" event={"ID":"15a0ed26-57d3-46d4-8535-82adda030709","Type":"ContainerDied","Data":"535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0"} Nov 02 20:05:10.834728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.834641 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l" event={"ID":"15a0ed26-57d3-46d4-8535-82adda030709","Type":"ContainerDied","Data":"6435650abfc8f772bedaaf9abbd04769160ca75e86269c6b85c38d6bedd24747"} Nov 02 20:05:10.834728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.834666 4966 scope.go:122] "RemoveContainer" containerID="535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0" Nov 02 20:05:10.836602 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.836572 4966 generic.go:356] "Generic (PLEG): container finished" podID="aa48ba08-be58-4f1c-8834-99b9c46b999f" containerID="a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76" exitCode=0 Nov 02 20:05:10.836729 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.836674 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" Nov 02 20:05:10.836729 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.836697 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" event={"ID":"aa48ba08-be58-4f1c-8834-99b9c46b999f","Type":"ContainerDied","Data":"a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76"} Nov 02 20:05:10.836821 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.836745 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl" event={"ID":"aa48ba08-be58-4f1c-8834-99b9c46b999f","Type":"ContainerDied","Data":"7380d8a94fbb27a18b4d053c74fd8171738618412e4a86d63b652c4a691acecc"} Nov 02 20:05:10.847783 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.847755 4966 scope.go:122] "RemoveContainer" containerID="535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0" Nov 02 20:05:10.848129 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:05:10.848106 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0\": container with ID starting with 535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0 not found: ID does not exist" containerID="535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0" Nov 02 20:05:10.848233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.848137 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0"} err="failed to get container status \"535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0\": rpc error: code = NotFound desc = could not find container \"535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0\": container with ID starting with 535fc4e5bce2b8cc040b3e8d58586679f1f5019a643650c11195c330404b4fb0 not found: ID does not exist" Nov 02 20:05:10.848233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.848155 4966 scope.go:122] "RemoveContainer" containerID="a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76" Nov 02 20:05:10.860137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.860068 4966 scope.go:122] "RemoveContainer" containerID="a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76" Nov 02 20:05:10.860450 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:05:10.860427 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76\": container with ID starting with a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76 not found: ID does not exist" containerID="a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76" Nov 02 20:05:10.860532 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.860457 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76"} err="failed to get container status \"a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76\": rpc error: code = NotFound desc = could not find container \"a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76\": container with ID starting with a88d07bb188de4414693c74e573b287b6222694894eca36c81fbc425b1cd1c76 not found: ID does not exist" Nov 02 20:05:10.990774 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.990738 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl"] Nov 02 20:05:10.995817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:10.995778 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c60a0d-66b66fd6d9-7ztrl"] Nov 02 20:05:11.094740 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:11.094660 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l"] Nov 02 20:05:11.098567 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:11.098541 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c60a0d-744ddb7487-z7n4l"] Nov 02 20:05:12.186152 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:12.186114 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a0ed26-57d3-46d4-8535-82adda030709" path="/var/lib/kubelet/pods/15a0ed26-57d3-46d4-8535-82adda030709/volumes" Nov 02 20:05:12.186639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:12.186623 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa48ba08-be58-4f1c-8834-99b9c46b999f" path="/var/lib/kubelet/pods/aa48ba08-be58-4f1c-8834-99b9c46b999f/volumes" Nov 02 20:05:14.758105 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:05:14.758066 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:05:17.578391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:17.578355 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf"] Nov 02 20:05:17.618047 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:17.618014 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32abda-qhjcf"] Nov 02 20:05:18.187229 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:18.187192 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37301b0f-5466-482e-a864-0bcde862b7bb" path="/var/lib/kubelet/pods/37301b0f-5466-482e-a864-0bcde862b7bb/volumes" Nov 02 20:05:18.574983 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:18.574940 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb"] Nov 02 20:05:18.618699 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:18.618669 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b0a12-mm9lb"] Nov 02 20:05:20.184783 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:20.184740 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1006eb85-72d7-4c65-b2b9-e06d14591848" path="/var/lib/kubelet/pods/1006eb85-72d7-4c65-b2b9-e06d14591848/volumes" Nov 02 20:05:25.021127 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.021089 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh"] Nov 02 20:05:25.021944 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.021921 4966 cpu_manager.go:414] "containerMap: removing container" podUID="aa48ba08-be58-4f1c-8834-99b9c46b999f" containerName="payload-processing" Nov 02 20:05:25.021944 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.021944 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa48ba08-be58-4f1c-8834-99b9c46b999f" containerName="payload-processing" Nov 02 20:05:25.022120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.021964 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4921acd7-9e9e-450c-8723-a13bac16040e" containerName="revoke-keys" Nov 02 20:05:25.022120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.021972 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4921acd7-9e9e-450c-8723-a13bac16040e" containerName="revoke-keys" Nov 02 20:05:25.022120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.022001 4966 cpu_manager.go:414] "containerMap: removing container" podUID="15a0ed26-57d3-46d4-8535-82adda030709" containerName="payload-pre-processing" Nov 02 20:05:25.022120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.022009 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a0ed26-57d3-46d4-8535-82adda030709" containerName="payload-pre-processing" Nov 02 20:05:25.022346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.022182 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa48ba08-be58-4f1c-8834-99b9c46b999f" containerName="payload-processing" Nov 02 20:05:25.022346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.022198 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4921acd7-9e9e-450c-8723-a13bac16040e" containerName="revoke-keys" Nov 02 20:05:25.022346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.022212 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a0ed26-57d3-46d4-8535-82adda030709" containerName="payload-pre-processing" Nov 02 20:05:25.027371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.027348 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.078061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.078027 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh"] Nov 02 20:05:25.119129 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.119090 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:05:25.119331 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.119144 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:05:25.119331 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.119231 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:05:25.121513 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.121496 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:05:25.130903 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.130878 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmk26\" (UniqueName: \"kubernetes.io/projected/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-kube-api-access-fmk26\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.131011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.130933 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.131123 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.131108 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.232707 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.232665 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmk26\" (UniqueName: \"kubernetes.io/projected/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-kube-api-access-fmk26\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.232927 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.232730 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.232927 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.232866 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.233542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.233515 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.235499 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.235476 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.285948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.285866 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmk26\" (UniqueName: \"kubernetes.io/projected/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-kube-api-access-fmk26\") pod \"maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.342110 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.342080 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:25.757016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:25.756985 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh"] Nov 02 20:05:25.757989 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:05:25.757957 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800d29f5_0c7d_4b35_9902_bc70a2df8cc7.slice/crio-4432f4d6ccac2605c785d6293f3fa8273982a5cf6df8b285a2564d09f3554734 WatchSource:0}: Error finding container 4432f4d6ccac2605c785d6293f3fa8273982a5cf6df8b285a2564d09f3554734: Status 404 returned error can't find the container with id 4432f4d6ccac2605c785d6293f3fa8273982a5cf6df8b285a2564d09f3554734 Nov 02 20:05:26.000228 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.000183 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" event={"ID":"800d29f5-0c7d-4b35-9902-bc70a2df8cc7","Type":"ContainerStarted","Data":"4432f4d6ccac2605c785d6293f3fa8273982a5cf6df8b285a2564d09f3554734"} Nov 02 20:05:26.540839 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.540802 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2"] Nov 02 20:05:26.545711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.545679 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.574430 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.574393 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.574430 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.574427 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhcv\" (UniqueName: \"kubernetes.io/projected/19992d69-4c32-4e6e-8499-4f5738f6bd0e-kube-api-access-vwhcv\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.574640 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.574577 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.598005 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.597976 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2"] Nov 02 20:05:26.675397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.675362 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.675588 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.675404 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhcv\" (UniqueName: \"kubernetes.io/projected/19992d69-4c32-4e6e-8499-4f5738f6bd0e-kube-api-access-vwhcv\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.675588 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.675507 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.676139 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.676112 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.677805 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.677775 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.727894 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.727859 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhcv\" (UniqueName: \"kubernetes.io/projected/19992d69-4c32-4e6e-8499-4f5738f6bd0e-kube-api-access-vwhcv\") pod \"maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:26.860634 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:26.860538 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:27.013443 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:27.013412 4966 generic.go:356] "Generic (PLEG): container finished" podID="800d29f5-0c7d-4b35-9902-bc70a2df8cc7" containerID="5e2013e1a341f2f35970ab718944d3a08bd97bbd11c7d96d7b7bcb99a1b6f091" exitCode=0 Nov 02 20:05:27.013619 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:27.013508 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" event={"ID":"800d29f5-0c7d-4b35-9902-bc70a2df8cc7","Type":"ContainerDied","Data":"5e2013e1a341f2f35970ab718944d3a08bd97bbd11c7d96d7b7bcb99a1b6f091"} Nov 02 20:05:27.238840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:27.238800 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2"] Nov 02 20:05:27.240629 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:05:27.240585 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19992d69_4c32_4e6e_8499_4f5738f6bd0e.slice/crio-f6a2164856175e15575c4afa5ee329f4d4dfd6164a000588ad1925f5704a7370 WatchSource:0}: Error finding container f6a2164856175e15575c4afa5ee329f4d4dfd6164a000588ad1925f5704a7370: Status 404 returned error can't find the container with id f6a2164856175e15575c4afa5ee329f4d4dfd6164a000588ad1925f5704a7370 Nov 02 20:05:28.028591 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.028549 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" event={"ID":"19992d69-4c32-4e6e-8499-4f5738f6bd0e","Type":"ContainerStarted","Data":"bc643b0076e1207847a3c24fb2e9ddf71766846fda6af5c4c0d3d39f9359fd0c"} Nov 02 20:05:28.029091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.028625 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" event={"ID":"19992d69-4c32-4e6e-8499-4f5738f6bd0e","Type":"ContainerStarted","Data":"f6a2164856175e15575c4afa5ee329f4d4dfd6164a000588ad1925f5704a7370"} Nov 02 20:05:28.154350 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.151421 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" podStartSLOduration=2.151401712 podStartE2EDuration="2.151401712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:28.151031769 +0000 UTC m=+2234.099687610" watchObservedRunningTime="2026-11-02 20:05:28.151401712 +0000 UTC m=+2234.100057552" Nov 02 20:05:28.248405 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.248377 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:28.325272 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.325240 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-xks-ca\") pod \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " Nov 02 20:05:28.325456 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.325359 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-kube-api-access-fmk26\" (UniqueName: \"kubernetes.io/projected/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-kube-api-access-fmk26\") pod \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " Nov 02 20:05:28.325529 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.325462 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-openshift-ca\") pod \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\" (UID: \"800d29f5-0c7d-4b35-9902-bc70a2df8cc7\") " Nov 02 20:05:28.325864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.325837 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-openshift-ca" pod "800d29f5-0c7d-4b35-9902-bc70a2df8cc7" (UID: "800d29f5-0c7d-4b35-9902-bc70a2df8cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:28.325986 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.325970 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:28.327260 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.327233 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-xks-ca" pod "800d29f5-0c7d-4b35-9902-bc70a2df8cc7" (UID: "800d29f5-0c7d-4b35-9902-bc70a2df8cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:28.327401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.327381 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-kube-api-access-fmk26" pod "800d29f5-0c7d-4b35-9902-bc70a2df8cc7" (UID: "800d29f5-0c7d-4b35-9902-bc70a2df8cc7"). InnerVolumeSpecName "kube-api-access-fmk26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:28.427451 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.427419 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:28.427451 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:28.427446 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmk26\" (UniqueName: \"kubernetes.io/projected/800d29f5-0c7d-4b35-9902-bc70a2df8cc7-kube-api-access-fmk26\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:29.037415 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:29.037382 4966 generic.go:356] "Generic (PLEG): container finished" podID="19992d69-4c32-4e6e-8499-4f5738f6bd0e" containerID="bc643b0076e1207847a3c24fb2e9ddf71766846fda6af5c4c0d3d39f9359fd0c" exitCode=0 Nov 02 20:05:29.037909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:29.037415 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" event={"ID":"19992d69-4c32-4e6e-8499-4f5738f6bd0e","Type":"ContainerDied","Data":"bc643b0076e1207847a3c24fb2e9ddf71766846fda6af5c4c0d3d39f9359fd0c"} Nov 02 20:05:29.038921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:29.038905 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" Nov 02 20:05:29.039095 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:29.038954 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh" event={"ID":"800d29f5-0c7d-4b35-9902-bc70a2df8cc7","Type":"ContainerDied","Data":"4432f4d6ccac2605c785d6293f3fa8273982a5cf6df8b285a2564d09f3554734"} Nov 02 20:05:29.039095 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:29.038997 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4432f4d6ccac2605c785d6293f3fa8273982a5cf6df8b285a2564d09f3554734" Nov 02 20:05:30.250580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.250549 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:30.357512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.357415 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19992d69-4c32-4e6e-8499-4f5738f6bd0e-kube-api-access-vwhcv\" (UniqueName: \"kubernetes.io/projected/19992d69-4c32-4e6e-8499-4f5738f6bd0e-kube-api-access-vwhcv\") pod \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " Nov 02 20:05:30.357512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.357499 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-openshift-ca\") pod \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " Nov 02 20:05:30.357731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.357558 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-xks-ca\") pod \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\" (UID: \"19992d69-4c32-4e6e-8499-4f5738f6bd0e\") " Nov 02 20:05:30.357859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.357829 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-openshift-ca" pod "19992d69-4c32-4e6e-8499-4f5738f6bd0e" (UID: "19992d69-4c32-4e6e-8499-4f5738f6bd0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:30.358263 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.358236 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:30.359965 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.359930 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-xks-ca" pod "19992d69-4c32-4e6e-8499-4f5738f6bd0e" (UID: "19992d69-4c32-4e6e-8499-4f5738f6bd0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:30.360090 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.360031 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19992d69-4c32-4e6e-8499-4f5738f6bd0e-kube-api-access-vwhcv" pod "19992d69-4c32-4e6e-8499-4f5738f6bd0e" (UID: "19992d69-4c32-4e6e-8499-4f5738f6bd0e"). InnerVolumeSpecName "kube-api-access-vwhcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:30.459958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.459917 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhcv\" (UniqueName: \"kubernetes.io/projected/19992d69-4c32-4e6e-8499-4f5738f6bd0e-kube-api-access-vwhcv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:30.459958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:30.459955 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19992d69-4c32-4e6e-8499-4f5738f6bd0e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:31.054846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:31.054675 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx"] Nov 02 20:05:31.062885 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:31.062862 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" Nov 02 20:05:31.063084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:31.062853 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2" event={"ID":"19992d69-4c32-4e6e-8499-4f5738f6bd0e","Type":"ContainerDied","Data":"f6a2164856175e15575c4afa5ee329f4d4dfd6164a000588ad1925f5704a7370"} Nov 02 20:05:31.063084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:31.062998 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a2164856175e15575c4afa5ee329f4d4dfd6164a000588ad1925f5704a7370" Nov 02 20:05:31.098504 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:31.098475 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8"] Nov 02 20:05:36.663659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:36.663623 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct"] Nov 02 20:05:39.263071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:39.263030 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98"] Nov 02 20:05:41.664214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:41.664154 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" podUID="bd2b3ee3-285e-4740-aee2-7003aa50baed" containerName="payload-processing" containerID="cri-o://15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a" gracePeriod=25 Nov 02 20:05:44.263448 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:44.263384 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" podUID="4f1f36d2-2a02-463d-b9ce-73265a4407af" containerName="payload-processing" containerID="cri-o://70aeeb92c6761c5a3a373938fc89490e4db0549e26ae9f71f1f45003b85eeca2" gracePeriod=25 Nov 02 20:05:44.753112 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:05:44.753074 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:05:45.091905 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:45.091823 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" podUID="4f1f36d2-2a02-463d-b9ce-73265a4407af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:46.081709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.081656 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" containerID="cri-o://e808a50e2856c3b9a55a92ae4fef4e945ce0d26a4430a360e54b270c2bf39442" gracePeriod=45 Nov 02 20:05:46.119581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.119543 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" containerID="cri-o://692c365279f3cd7ae4037b470ece5c68e04cf86588ff70eaea2d487d9c8fcc91" gracePeriod=45 Nov 02 20:05:46.140668 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.140621 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused" Nov 02 20:05:46.301205 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.301166 4966 generic.go:356] "Generic (PLEG): container finished" podID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerID="e808a50e2856c3b9a55a92ae4fef4e945ce0d26a4430a360e54b270c2bf39442" exitCode=0 Nov 02 20:05:46.301403 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.301216 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" event={"ID":"c8801ef5-65bb-4798-a12c-6f769c84ba1a","Type":"ContainerDied","Data":"e808a50e2856c3b9a55a92ae4fef4e945ce0d26a4430a360e54b270c2bf39442"} Nov 02 20:05:46.303831 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.303796 4966 generic.go:356] "Generic (PLEG): container finished" podID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerID="692c365279f3cd7ae4037b470ece5c68e04cf86588ff70eaea2d487d9c8fcc91" exitCode=0 Nov 02 20:05:46.304003 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.303980 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" event={"ID":"b698a18c-5871-4d0c-8ca3-387fcee7e0e1","Type":"ContainerDied","Data":"692c365279f3cd7ae4037b470ece5c68e04cf86588ff70eaea2d487d9c8fcc91"} Nov 02 20:05:46.391892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.391856 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 20:05:46.451656 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.451624 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 20:05:46.490951 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.490913 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-dshm\") pod \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " Nov 02 20:05:46.491171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.490958 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8801ef5-65bb-4798-a12c-6f769c84ba1a-kube-api-access-dvh5j\" (UniqueName: \"kubernetes.io/projected/c8801ef5-65bb-4798-a12c-6f769c84ba1a-kube-api-access-dvh5j\") pod \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " Nov 02 20:05:46.491171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.490983 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-kube-api-access-gpk4g\" (UniqueName: \"kubernetes.io/projected/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-kube-api-access-gpk4g\") pod \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " Nov 02 20:05:46.491171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491012 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-model-cache\") pod \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " Nov 02 20:05:46.491171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491061 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tmp-dir\") pod \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " Nov 02 20:05:46.491171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491099 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-home\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-home\") pod \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " Nov 02 20:05:46.491171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491135 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tmp-dir\") pod \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " Nov 02 20:05:46.491580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491182 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-dshm\") pod \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " Nov 02 20:05:46.491580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491234 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-home\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-home\") pod \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " Nov 02 20:05:46.491580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491275 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-model-cache\") pod \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " Nov 02 20:05:46.491580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491339 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tls-certs\") pod \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\" (UID: \"b698a18c-5871-4d0c-8ca3-387fcee7e0e1\") " Nov 02 20:05:46.491580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491359 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tmp-dir" pod "c8801ef5-65bb-4798-a12c-6f769c84ba1a" (UID: "c8801ef5-65bb-4798-a12c-6f769c84ba1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.491580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491394 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-model-cache" pod "b698a18c-5871-4d0c-8ca3-387fcee7e0e1" (UID: "b698a18c-5871-4d0c-8ca3-387fcee7e0e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.491580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491437 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tls-certs\") pod \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\" (UID: \"c8801ef5-65bb-4798-a12c-6f769c84ba1a\") " Nov 02 20:05:46.491971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491610 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-home" pod "c8801ef5-65bb-4798-a12c-6f769c84ba1a" (UID: "c8801ef5-65bb-4798-a12c-6f769c84ba1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.491971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491729 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-home" pod "b698a18c-5871-4d0c-8ca3-387fcee7e0e1" (UID: "b698a18c-5871-4d0c-8ca3-387fcee7e0e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.491971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491824 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-model-cache" pod "c8801ef5-65bb-4798-a12c-6f769c84ba1a" (UID: "c8801ef5-65bb-4798-a12c-6f769c84ba1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.491971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.491951 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tmp-dir" pod "b698a18c-5871-4d0c-8ca3-387fcee7e0e1" (UID: "b698a18c-5871-4d0c-8ca3-387fcee7e0e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.492200 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.492174 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.492200 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.492195 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.492343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.492210 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.492343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.492224 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.492343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.492236 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.492343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.492250 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.493656 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.493626 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-dshm" pod "b698a18c-5871-4d0c-8ca3-387fcee7e0e1" (UID: "b698a18c-5871-4d0c-8ca3-387fcee7e0e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.493799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.493730 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tls-certs" pod "b698a18c-5871-4d0c-8ca3-387fcee7e0e1" (UID: "b698a18c-5871-4d0c-8ca3-387fcee7e0e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:46.493855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.493797 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-kube-api-access-gpk4g" pod "b698a18c-5871-4d0c-8ca3-387fcee7e0e1" (UID: "b698a18c-5871-4d0c-8ca3-387fcee7e0e1"). InnerVolumeSpecName "kube-api-access-gpk4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:46.493970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.493949 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8801ef5-65bb-4798-a12c-6f769c84ba1a-kube-api-access-dvh5j" pod "c8801ef5-65bb-4798-a12c-6f769c84ba1a" (UID: "c8801ef5-65bb-4798-a12c-6f769c84ba1a"). InnerVolumeSpecName "kube-api-access-dvh5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:46.494079 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.494006 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-dshm" pod "c8801ef5-65bb-4798-a12c-6f769c84ba1a" (UID: "c8801ef5-65bb-4798-a12c-6f769c84ba1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:05:46.494121 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.494107 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tls-certs" pod "c8801ef5-65bb-4798-a12c-6f769c84ba1a" (UID: "c8801ef5-65bb-4798-a12c-6f769c84ba1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:46.593744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.593700 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8801ef5-65bb-4798-a12c-6f769c84ba1a-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.593744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.593737 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.594021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.593753 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8801ef5-65bb-4798-a12c-6f769c84ba1a-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.594021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.593769 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.594021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.593791 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvh5j\" (UniqueName: \"kubernetes.io/projected/c8801ef5-65bb-4798-a12c-6f769c84ba1a-kube-api-access-dvh5j\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:46.594021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:46.593807 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpk4g\" (UniqueName: \"kubernetes.io/projected/b698a18c-5871-4d0c-8ca3-387fcee7e0e1-kube-api-access-gpk4g\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:47.301692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.301649 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:05:47.314945 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.314918 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" Nov 02 20:05:47.315130 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.314942 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx" event={"ID":"c8801ef5-65bb-4798-a12c-6f769c84ba1a","Type":"ContainerDied","Data":"3f67e5969e26361372e593ad298642d23a5384e009e753365b8ac8f3e1f12e0a"} Nov 02 20:05:47.315130 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.314994 4966 scope.go:122] "RemoveContainer" containerID="e808a50e2856c3b9a55a92ae4fef4e945ce0d26a4430a360e54b270c2bf39442" Nov 02 20:05:47.316711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.316686 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" event={"ID":"b698a18c-5871-4d0c-8ca3-387fcee7e0e1","Type":"ContainerDied","Data":"cc75b90fa2cab8374e4259672fe9be7af5006e6376ed64d0b171e42f1b4238f0"} Nov 02 20:05:47.316794 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.316772 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8" Nov 02 20:05:47.329976 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.329949 4966 scope.go:122] "RemoveContainer" containerID="692c365279f3cd7ae4037b470ece5c68e04cf86588ff70eaea2d487d9c8fcc91" Nov 02 20:05:47.476400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.476351 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx"] Nov 02 20:05:47.478369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.478337 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/distinct-2-w2-28623b-kserve-5dd76f9fc5-jnhcx"] Nov 02 20:05:47.575096 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.575055 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8"] Nov 02 20:05:47.581426 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.581396 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-28623b/simulator-w2-28623b-kserve-7bd9df4cdd-rr2z8"] Nov 02 20:05:47.948838 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:47.948740 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" podUID="bd2b3ee3-285e-4740-aee2-7003aa50baed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:48.184342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:48.184285 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" path="/var/lib/kubelet/pods/b698a18c-5871-4d0c-8ca3-387fcee7e0e1/volumes" Nov 02 20:05:48.184819 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:48.184801 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" path="/var/lib/kubelet/pods/c8801ef5-65bb-4798-a12c-6f769c84ba1a/volumes" Nov 02 20:05:49.933731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.933692 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg"] Nov 02 20:05:49.934659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934631 4966 cpu_manager.go:414] "containerMap: removing container" podUID="800d29f5-0c7d-4b35-9902-bc70a2df8cc7" containerName="revoke-keys" Nov 02 20:05:49.934659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934658 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800d29f5-0c7d-4b35-9902-bc70a2df8cc7" containerName="revoke-keys" Nov 02 20:05:49.934859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934680 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" Nov 02 20:05:49.934859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934688 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" Nov 02 20:05:49.934859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934697 4966 cpu_manager.go:414] "containerMap: removing container" podUID="19992d69-4c32-4e6e-8499-4f5738f6bd0e" containerName="revoke-keys" Nov 02 20:05:49.934859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934704 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19992d69-4c32-4e6e-8499-4f5738f6bd0e" containerName="revoke-keys" Nov 02 20:05:49.934859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934732 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" Nov 02 20:05:49.934859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934739 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" Nov 02 20:05:49.935078 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934948 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b698a18c-5871-4d0c-8ca3-387fcee7e0e1" containerName="main" Nov 02 20:05:49.935078 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934976 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="800d29f5-0c7d-4b35-9902-bc70a2df8cc7" containerName="revoke-keys" Nov 02 20:05:49.935078 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934988 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8801ef5-65bb-4798-a12c-6f769c84ba1a" containerName="main" Nov 02 20:05:49.935078 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.934998 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="19992d69-4c32-4e6e-8499-4f5738f6bd0e" containerName="revoke-keys" Nov 02 20:05:49.940481 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.940460 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:49.987461 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.987429 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg"] Nov 02 20:05:49.991901 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.991877 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg"] Nov 02 20:05:49.992046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:49.991965 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.029342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.029274 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:05:50.029525 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.029342 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:05:50.029525 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.029341 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:05:50.031802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.031776 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:05:50.045260 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.045228 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg"] Nov 02 20:05:50.049948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.049720 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7xh\" (UniqueName: \"kubernetes.io/projected/392c2aac-340b-4a15-93b2-f3ff7179f44d-kube-api-access-xh7xh\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.049948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.049920 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.050174 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.049975 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.050174 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.050135 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mhz7\" (UniqueName: \"kubernetes.io/projected/d7b8a94f-a0a3-4387-82af-d70c20d0d604-kube-api-access-2mhz7\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.050333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.050177 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.050333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.050203 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.152103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.152061 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.152349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.152144 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.152349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.152250 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mhz7\" (UniqueName: \"kubernetes.io/projected/d7b8a94f-a0a3-4387-82af-d70c20d0d604-kube-api-access-2mhz7\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.152349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.152292 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.152540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.152354 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.152540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.152393 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7xh\" (UniqueName: \"kubernetes.io/projected/392c2aac-340b-4a15-93b2-f3ff7179f44d-kube-api-access-xh7xh\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.153062 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.152916 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.153217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.153062 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.155203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.155175 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.155203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.155182 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.206450 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.206365 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mhz7\" (UniqueName: \"kubernetes.io/projected/d7b8a94f-a0a3-4387-82af-d70c20d0d604-kube-api-access-2mhz7\") pod \"maas-api-revoke-sub-2362d11a-6dcfg\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.216219 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.216183 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7xh\" (UniqueName: \"kubernetes.io/projected/392c2aac-340b-4a15-93b2-f3ff7179f44d-kube-api-access-xh7xh\") pod \"maas-api-revoke-sub-59461aa9-rr4fg\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.253961 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.253926 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:50.312495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.310118 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:50.626734 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.626701 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg"] Nov 02 20:05:50.627033 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:05:50.627004 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b8a94f_a0a3_4387_82af_d70c20d0d604.slice/crio-86bb46b03767d28ab3e761207f90ea3f806ccc8f023050fed4550e80dee27c17 WatchSource:0}: Error finding container 86bb46b03767d28ab3e761207f90ea3f806ccc8f023050fed4550e80dee27c17: Status 404 returned error can't find the container with id 86bb46b03767d28ab3e761207f90ea3f806ccc8f023050fed4550e80dee27c17 Nov 02 20:05:50.730000 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:50.729966 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg"] Nov 02 20:05:50.730443 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:05:50.730413 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392c2aac_340b_4a15_93b2_f3ff7179f44d.slice/crio-fdccef2e32f23fe8252acdd7a3a8793d102b8d40f320a76d611238c400a78f81 WatchSource:0}: Error finding container fdccef2e32f23fe8252acdd7a3a8793d102b8d40f320a76d611238c400a78f81: Status 404 returned error can't find the container with id fdccef2e32f23fe8252acdd7a3a8793d102b8d40f320a76d611238c400a78f81 Nov 02 20:05:51.359372 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:51.359262 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" event={"ID":"d7b8a94f-a0a3-4387-82af-d70c20d0d604","Type":"ContainerStarted","Data":"4edd0129c9a8b9f82dd2f317c02ec98f3a7ba9d90d8e0e4a397667484c6249e1"} Nov 02 20:05:51.359372 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:51.359364 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" event={"ID":"d7b8a94f-a0a3-4387-82af-d70c20d0d604","Type":"ContainerStarted","Data":"86bb46b03767d28ab3e761207f90ea3f806ccc8f023050fed4550e80dee27c17"} Nov 02 20:05:51.361675 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:51.361643 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" event={"ID":"392c2aac-340b-4a15-93b2-f3ff7179f44d","Type":"ContainerStarted","Data":"ff26f95e866129d7e2fab8f9c9711b667b61f80a2e77f308352d781fd58e4a00"} Nov 02 20:05:51.361802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:51.361697 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" event={"ID":"392c2aac-340b-4a15-93b2-f3ff7179f44d","Type":"ContainerStarted","Data":"fdccef2e32f23fe8252acdd7a3a8793d102b8d40f320a76d611238c400a78f81"} Nov 02 20:05:51.466453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:51.466393 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" podStartSLOduration=2.466374139 podStartE2EDuration="2.466374139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:51.462721122 +0000 UTC m=+2257.411376960" watchObservedRunningTime="2026-11-02 20:05:51.466374139 +0000 UTC m=+2257.415029979" Nov 02 20:05:51.569700 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:51.569642 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" podStartSLOduration=2.569621684 podStartE2EDuration="2.569621684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:51.566920981 +0000 UTC m=+2257.515576822" watchObservedRunningTime="2026-11-02 20:05:51.569621684 +0000 UTC m=+2257.518277517" Nov 02 20:05:52.370991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:52.370956 4966 generic.go:356] "Generic (PLEG): container finished" podID="392c2aac-340b-4a15-93b2-f3ff7179f44d" containerID="ff26f95e866129d7e2fab8f9c9711b667b61f80a2e77f308352d781fd58e4a00" exitCode=0 Nov 02 20:05:52.371457 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:52.371041 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" event={"ID":"392c2aac-340b-4a15-93b2-f3ff7179f44d","Type":"ContainerDied","Data":"ff26f95e866129d7e2fab8f9c9711b667b61f80a2e77f308352d781fd58e4a00"} Nov 02 20:05:52.372827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:52.372803 4966 generic.go:356] "Generic (PLEG): container finished" podID="d7b8a94f-a0a3-4387-82af-d70c20d0d604" containerID="4edd0129c9a8b9f82dd2f317c02ec98f3a7ba9d90d8e0e4a397667484c6249e1" exitCode=0 Nov 02 20:05:52.372972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:52.372860 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" event={"ID":"d7b8a94f-a0a3-4387-82af-d70c20d0d604","Type":"ContainerDied","Data":"4edd0129c9a8b9f82dd2f317c02ec98f3a7ba9d90d8e0e4a397667484c6249e1"} Nov 02 20:05:53.553262 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.553231 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:53.594462 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.594437 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:53.602051 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.602028 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b8a94f-a0a3-4387-82af-d70c20d0d604-kube-api-access-2mhz7\" (UniqueName: \"kubernetes.io/projected/d7b8a94f-a0a3-4387-82af-d70c20d0d604-kube-api-access-2mhz7\") pod \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " Nov 02 20:05:53.602162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.602076 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-xks-ca\") pod \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " Nov 02 20:05:53.602208 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.602198 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-openshift-ca\") pod \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\" (UID: \"d7b8a94f-a0a3-4387-82af-d70c20d0d604\") " Nov 02 20:05:53.602615 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.602590 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-openshift-ca" pod "d7b8a94f-a0a3-4387-82af-d70c20d0d604" (UID: "d7b8a94f-a0a3-4387-82af-d70c20d0d604"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:53.602807 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.602781 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:53.604159 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.604135 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-xks-ca" pod "d7b8a94f-a0a3-4387-82af-d70c20d0d604" (UID: "d7b8a94f-a0a3-4387-82af-d70c20d0d604"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:53.604263 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.604233 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b8a94f-a0a3-4387-82af-d70c20d0d604-kube-api-access-2mhz7" pod "d7b8a94f-a0a3-4387-82af-d70c20d0d604" (UID: "d7b8a94f-a0a3-4387-82af-d70c20d0d604"). InnerVolumeSpecName "kube-api-access-2mhz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:53.703512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.703408 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-openshift-ca\") pod \"392c2aac-340b-4a15-93b2-f3ff7179f44d\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " Nov 02 20:05:53.703697 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.703604 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-xks-ca\") pod \"392c2aac-340b-4a15-93b2-f3ff7179f44d\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " Nov 02 20:05:53.703783 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.703765 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392c2aac-340b-4a15-93b2-f3ff7179f44d-kube-api-access-xh7xh\" (UniqueName: \"kubernetes.io/projected/392c2aac-340b-4a15-93b2-f3ff7179f44d-kube-api-access-xh7xh\") pod \"392c2aac-340b-4a15-93b2-f3ff7179f44d\" (UID: \"392c2aac-340b-4a15-93b2-f3ff7179f44d\") " Nov 02 20:05:53.703848 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.703823 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-openshift-ca" pod "392c2aac-340b-4a15-93b2-f3ff7179f44d" (UID: "392c2aac-340b-4a15-93b2-f3ff7179f44d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:53.704391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.704356 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:53.704391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.704386 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mhz7\" (UniqueName: \"kubernetes.io/projected/d7b8a94f-a0a3-4387-82af-d70c20d0d604-kube-api-access-2mhz7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:53.704666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.704403 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7b8a94f-a0a3-4387-82af-d70c20d0d604-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:53.706611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.706580 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-xks-ca" pod "392c2aac-340b-4a15-93b2-f3ff7179f44d" (UID: "392c2aac-340b-4a15-93b2-f3ff7179f44d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:05:53.706841 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.706814 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392c2aac-340b-4a15-93b2-f3ff7179f44d-kube-api-access-xh7xh" pod "392c2aac-340b-4a15-93b2-f3ff7179f44d" (UID: "392c2aac-340b-4a15-93b2-f3ff7179f44d"). InnerVolumeSpecName "kube-api-access-xh7xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:53.805532 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.805492 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392c2aac-340b-4a15-93b2-f3ff7179f44d-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:53.805532 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:53.805529 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7xh\" (UniqueName: \"kubernetes.io/projected/392c2aac-340b-4a15-93b2-f3ff7179f44d-kube-api-access-xh7xh\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:54.392225 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:54.392193 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" Nov 02 20:05:54.392225 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:54.392214 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg" event={"ID":"d7b8a94f-a0a3-4387-82af-d70c20d0d604","Type":"ContainerDied","Data":"86bb46b03767d28ab3e761207f90ea3f806ccc8f023050fed4550e80dee27c17"} Nov 02 20:05:54.392485 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:54.392275 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86bb46b03767d28ab3e761207f90ea3f806ccc8f023050fed4550e80dee27c17" Nov 02 20:05:54.394135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:54.394108 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" event={"ID":"392c2aac-340b-4a15-93b2-f3ff7179f44d","Type":"ContainerDied","Data":"fdccef2e32f23fe8252acdd7a3a8793d102b8d40f320a76d611238c400a78f81"} Nov 02 20:05:54.394284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:54.394138 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg" Nov 02 20:05:54.394284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:54.394153 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdccef2e32f23fe8252acdd7a3a8793d102b8d40f320a76d611238c400a78f81" Nov 02 20:05:55.092521 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:55.092292 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" podUID="4f1f36d2-2a02-463d-b9ce-73265a4407af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:57.948387 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:57.948343 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" podUID="bd2b3ee3-285e-4740-aee2-7003aa50baed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:58.924582 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.924540 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g"] Nov 02 20:05:58.925525 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.925505 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d7b8a94f-a0a3-4387-82af-d70c20d0d604" containerName="revoke-keys" Nov 02 20:05:58.925581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.925528 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b8a94f-a0a3-4387-82af-d70c20d0d604" containerName="revoke-keys" Nov 02 20:05:58.925581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.925540 4966 cpu_manager.go:414] "containerMap: removing container" podUID="392c2aac-340b-4a15-93b2-f3ff7179f44d" containerName="revoke-keys" Nov 02 20:05:58.925581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.925547 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392c2aac-340b-4a15-93b2-f3ff7179f44d" containerName="revoke-keys" Nov 02 20:05:58.925712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.925698 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b8a94f-a0a3-4387-82af-d70c20d0d604" containerName="revoke-keys" Nov 02 20:05:58.925748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.925719 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="392c2aac-340b-4a15-93b2-f3ff7179f44d" containerName="revoke-keys" Nov 02 20:05:58.930939 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.930914 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:58.976599 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.976559 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj"] Nov 02 20:05:58.981524 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.981500 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g"] Nov 02 20:05:58.981694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:58.981619 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.018120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.018078 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:05:59.018395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.018117 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:05:59.018395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.018142 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:05:59.021836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.021808 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:05:59.036072 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.036038 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj"] Nov 02 20:05:59.089321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.089208 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.089533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.089383 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.089533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.089463 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjd77\" (UniqueName: \"kubernetes.io/projected/285f6908-ff18-4259-b37f-b0a4ea9acb64-kube-api-access-cjd77\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.089677 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.089541 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.089677 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.089565 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4chd\" (UniqueName: \"kubernetes.io/projected/989c00b9-29a1-448d-81cf-7e76c5b254ee-kube-api-access-d4chd\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.089794 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.089671 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.191248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.191156 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.191248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.191209 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.191537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.191256 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjd77\" (UniqueName: \"kubernetes.io/projected/285f6908-ff18-4259-b37f-b0a4ea9acb64-kube-api-access-cjd77\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.191537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.191348 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.191537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.191375 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4chd\" (UniqueName: \"kubernetes.io/projected/989c00b9-29a1-448d-81cf-7e76c5b254ee-kube-api-access-d4chd\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.191537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.191398 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.192101 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.192072 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.192289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.192265 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.193896 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.193867 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.193991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.193924 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.248562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.246600 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjd77\" (UniqueName: \"kubernetes.io/projected/285f6908-ff18-4259-b37f-b0a4ea9acb64-kube-api-access-cjd77\") pod \"maas-api-revoke-sub-f22e7c9c-lmqlj\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.253869 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.253835 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4chd\" (UniqueName: \"kubernetes.io/projected/989c00b9-29a1-448d-81cf-7e76c5b254ee-kube-api-access-d4chd\") pod \"maas-api-revoke-sub-fd1ff980-ldx8g\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.299921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.299885 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:05:59.548598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.547629 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:05:59.750698 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.750666 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj"] Nov 02 20:05:59.751187 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:05:59.751168 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285f6908_ff18_4259_b37f_b0a4ea9acb64.slice/crio-4a00a6cd2b3913da54de9b854805e8be5dc15de45cac864c5cdb3e32954f1ec2 WatchSource:0}: Error finding container 4a00a6cd2b3913da54de9b854805e8be5dc15de45cac864c5cdb3e32954f1ec2: Status 404 returned error can't find the container with id 4a00a6cd2b3913da54de9b854805e8be5dc15de45cac864c5cdb3e32954f1ec2 Nov 02 20:05:59.966527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:05:59.966489 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g"] Nov 02 20:06:00.480003 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:00.479699 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" event={"ID":"989c00b9-29a1-448d-81cf-7e76c5b254ee","Type":"ContainerStarted","Data":"591ceb63830c1ea4b0076763ce045b9866951ca95bedeba031774479a68120c0"} Nov 02 20:06:00.480003 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:00.479772 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" event={"ID":"989c00b9-29a1-448d-81cf-7e76c5b254ee","Type":"ContainerStarted","Data":"8bbdbe130590bf2c3834076e0e52e1e5f8c1ec8fe753e91608417be97fa00c12"} Nov 02 20:06:00.483658 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:00.483622 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" event={"ID":"285f6908-ff18-4259-b37f-b0a4ea9acb64","Type":"ContainerStarted","Data":"cf553e7fa33c5a7371eb5d19c440036f8e9969faf8bef93801da7dad34f4b133"} Nov 02 20:06:00.483796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:00.483676 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" event={"ID":"285f6908-ff18-4259-b37f-b0a4ea9acb64","Type":"ContainerStarted","Data":"4a00a6cd2b3913da54de9b854805e8be5dc15de45cac864c5cdb3e32954f1ec2"} Nov 02 20:06:00.587078 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:00.586970 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" podStartSLOduration=2.586953721 podStartE2EDuration="2.586953721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:00.58313792 +0000 UTC m=+2266.531793758" watchObservedRunningTime="2026-11-02 20:06:00.586953721 +0000 UTC m=+2266.535609560" Nov 02 20:06:00.691956 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:00.691900 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" podStartSLOduration=2.69188691 podStartE2EDuration="2.69188691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:00.687796487 +0000 UTC m=+2266.636452327" watchObservedRunningTime="2026-11-02 20:06:00.69188691 +0000 UTC m=+2266.640542750" Nov 02 20:06:01.214528 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:01.214492 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz"] Nov 02 20:06:01.219442 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:01.219412 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c60a0d-h6kgz"] Nov 02 20:06:01.494631 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:01.494598 4966 generic.go:356] "Generic (PLEG): container finished" podID="989c00b9-29a1-448d-81cf-7e76c5b254ee" containerID="591ceb63830c1ea4b0076763ce045b9866951ca95bedeba031774479a68120c0" exitCode=0 Nov 02 20:06:01.495109 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:01.494691 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" event={"ID":"989c00b9-29a1-448d-81cf-7e76c5b254ee","Type":"ContainerDied","Data":"591ceb63830c1ea4b0076763ce045b9866951ca95bedeba031774479a68120c0"} Nov 02 20:06:01.496358 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:01.496334 4966 generic.go:356] "Generic (PLEG): container finished" podID="285f6908-ff18-4259-b37f-b0a4ea9acb64" containerID="cf553e7fa33c5a7371eb5d19c440036f8e9969faf8bef93801da7dad34f4b133" exitCode=0 Nov 02 20:06:01.496499 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:01.496413 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" event={"ID":"285f6908-ff18-4259-b37f-b0a4ea9acb64","Type":"ContainerDied","Data":"cf553e7fa33c5a7371eb5d19c440036f8e9969faf8bef93801da7dad34f4b133"} Nov 02 20:06:01.956737 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:01.956712 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 20:06:02.075525 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.075435 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2b3ee3-285e-4740-aee2-7003aa50baed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2b3ee3-285e-4740-aee2-7003aa50baed-plugins-config-volume\") pod \"bd2b3ee3-285e-4740-aee2-7003aa50baed\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " Nov 02 20:06:02.075713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.075533 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2b3ee3-285e-4740-aee2-7003aa50baed-kube-api-access-6m55t\" (UniqueName: \"kubernetes.io/projected/bd2b3ee3-285e-4740-aee2-7003aa50baed-kube-api-access-6m55t\") pod \"bd2b3ee3-285e-4740-aee2-7003aa50baed\" (UID: \"bd2b3ee3-285e-4740-aee2-7003aa50baed\") " Nov 02 20:06:02.076280 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.076252 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2b3ee3-285e-4740-aee2-7003aa50baed-plugins-config-volume" pod "bd2b3ee3-285e-4740-aee2-7003aa50baed" (UID: "bd2b3ee3-285e-4740-aee2-7003aa50baed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:02.083862 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.083828 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2b3ee3-285e-4740-aee2-7003aa50baed-kube-api-access-6m55t" pod "bd2b3ee3-285e-4740-aee2-7003aa50baed" (UID: "bd2b3ee3-285e-4740-aee2-7003aa50baed"). InnerVolumeSpecName "kube-api-access-6m55t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:02.177314 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.177266 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2b3ee3-285e-4740-aee2-7003aa50baed-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:02.177314 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.177323 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m55t\" (UniqueName: \"kubernetes.io/projected/bd2b3ee3-285e-4740-aee2-7003aa50baed-kube-api-access-6m55t\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:02.186452 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.186411 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4921acd7-9e9e-450c-8723-a13bac16040e" path="/var/lib/kubelet/pods/4921acd7-9e9e-450c-8723-a13bac16040e/volumes" Nov 02 20:06:02.505952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.505913 4966 generic.go:356] "Generic (PLEG): container finished" podID="bd2b3ee3-285e-4740-aee2-7003aa50baed" containerID="15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a" exitCode=0 Nov 02 20:06:02.506461 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.505958 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" Nov 02 20:06:02.506461 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.506001 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" event={"ID":"bd2b3ee3-285e-4740-aee2-7003aa50baed","Type":"ContainerDied","Data":"15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a"} Nov 02 20:06:02.506461 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.506054 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct" event={"ID":"bd2b3ee3-285e-4740-aee2-7003aa50baed","Type":"ContainerDied","Data":"8f2987d0fa13ee912ffb768189ad625787d6a70b3725615ea60adfb99588c9b4"} Nov 02 20:06:02.506461 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.506079 4966 scope.go:122] "RemoveContainer" containerID="15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a" Nov 02 20:06:02.517795 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.517761 4966 scope.go:122] "RemoveContainer" containerID="15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a" Nov 02 20:06:02.518176 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:06:02.518143 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a\": container with ID starting with 15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a not found: ID does not exist" containerID="15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a" Nov 02 20:06:02.518257 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.518196 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a"} err="failed to get container status \"15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a\": rpc error: code = NotFound desc = could not find container \"15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a\": container with ID starting with 15fdcc47b04edf025b1d69d932485f8caea4457ce75f3a243c97b6d8db96c17a not found: ID does not exist" Nov 02 20:06:02.665331 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.664341 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct"] Nov 02 20:06:02.706903 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.706872 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-1d869e-d9d949bcf-hk9ct"] Nov 02 20:06:02.718008 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.717985 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:06:02.720060 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.720041 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:06:02.884787 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.884695 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-openshift-ca\") pod \"989c00b9-29a1-448d-81cf-7e76c5b254ee\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " Nov 02 20:06:02.884787 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.884751 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-xks-ca\") pod \"285f6908-ff18-4259-b37f-b0a4ea9acb64\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " Nov 02 20:06:02.884787 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.884773 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285f6908-ff18-4259-b37f-b0a4ea9acb64-kube-api-access-cjd77\" (UniqueName: \"kubernetes.io/projected/285f6908-ff18-4259-b37f-b0a4ea9acb64-kube-api-access-cjd77\") pod \"285f6908-ff18-4259-b37f-b0a4ea9acb64\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " Nov 02 20:06:02.885105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.884833 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-xks-ca\") pod \"989c00b9-29a1-448d-81cf-7e76c5b254ee\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " Nov 02 20:06:02.885105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.884909 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-openshift-ca\") pod \"285f6908-ff18-4259-b37f-b0a4ea9acb64\" (UID: \"285f6908-ff18-4259-b37f-b0a4ea9acb64\") " Nov 02 20:06:02.885105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.885098 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989c00b9-29a1-448d-81cf-7e76c5b254ee-kube-api-access-d4chd\" (UniqueName: \"kubernetes.io/projected/989c00b9-29a1-448d-81cf-7e76c5b254ee-kube-api-access-d4chd\") pod \"989c00b9-29a1-448d-81cf-7e76c5b254ee\" (UID: \"989c00b9-29a1-448d-81cf-7e76c5b254ee\") " Nov 02 20:06:02.885286 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.885159 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-openshift-ca" pod "989c00b9-29a1-448d-81cf-7e76c5b254ee" (UID: "989c00b9-29a1-448d-81cf-7e76c5b254ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:02.885561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.885538 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-openshift-ca" pod "285f6908-ff18-4259-b37f-b0a4ea9acb64" (UID: "285f6908-ff18-4259-b37f-b0a4ea9acb64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:02.885891 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.885736 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:02.885891 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.885755 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:02.887423 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.887399 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285f6908-ff18-4259-b37f-b0a4ea9acb64-kube-api-access-cjd77" pod "285f6908-ff18-4259-b37f-b0a4ea9acb64" (UID: "285f6908-ff18-4259-b37f-b0a4ea9acb64"). InnerVolumeSpecName "kube-api-access-cjd77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:02.887616 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.887591 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-xks-ca" pod "989c00b9-29a1-448d-81cf-7e76c5b254ee" (UID: "989c00b9-29a1-448d-81cf-7e76c5b254ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:06:02.887733 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.887650 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989c00b9-29a1-448d-81cf-7e76c5b254ee-kube-api-access-d4chd" pod "989c00b9-29a1-448d-81cf-7e76c5b254ee" (UID: "989c00b9-29a1-448d-81cf-7e76c5b254ee"). InnerVolumeSpecName "kube-api-access-d4chd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:02.887733 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.887674 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-xks-ca" pod "285f6908-ff18-4259-b37f-b0a4ea9acb64" (UID: "285f6908-ff18-4259-b37f-b0a4ea9acb64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:06:02.987333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.987282 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4chd\" (UniqueName: \"kubernetes.io/projected/989c00b9-29a1-448d-81cf-7e76c5b254ee-kube-api-access-d4chd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:02.987333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.987331 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285f6908-ff18-4259-b37f-b0a4ea9acb64-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:02.987333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.987343 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjd77\" (UniqueName: \"kubernetes.io/projected/285f6908-ff18-4259-b37f-b0a4ea9acb64-kube-api-access-cjd77\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:02.987572 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:02.987352 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989c00b9-29a1-448d-81cf-7e76c5b254ee-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:03.517061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:03.517028 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" Nov 02 20:06:03.517061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:03.517048 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g" event={"ID":"989c00b9-29a1-448d-81cf-7e76c5b254ee","Type":"ContainerDied","Data":"8bbdbe130590bf2c3834076e0e52e1e5f8c1ec8fe753e91608417be97fa00c12"} Nov 02 20:06:03.517616 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:03.517096 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bbdbe130590bf2c3834076e0e52e1e5f8c1ec8fe753e91608417be97fa00c12" Nov 02 20:06:03.518480 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:03.518457 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" event={"ID":"285f6908-ff18-4259-b37f-b0a4ea9acb64","Type":"ContainerDied","Data":"4a00a6cd2b3913da54de9b854805e8be5dc15de45cac864c5cdb3e32954f1ec2"} Nov 02 20:06:03.518573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:03.518494 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a00a6cd2b3913da54de9b854805e8be5dc15de45cac864c5cdb3e32954f1ec2" Nov 02 20:06:03.518573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:03.518496 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj" Nov 02 20:06:04.185608 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.185565 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2b3ee3-285e-4740-aee2-7003aa50baed" path="/var/lib/kubelet/pods/bd2b3ee3-285e-4740-aee2-7003aa50baed/volumes" Nov 02 20:06:04.532631 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.532602 4966 generic.go:356] "Generic (PLEG): container finished" podID="4f1f36d2-2a02-463d-b9ce-73265a4407af" containerID="70aeeb92c6761c5a3a373938fc89490e4db0549e26ae9f71f1f45003b85eeca2" exitCode=0 Nov 02 20:06:04.532955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.532674 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" event={"ID":"4f1f36d2-2a02-463d-b9ce-73265a4407af","Type":"ContainerDied","Data":"70aeeb92c6761c5a3a373938fc89490e4db0549e26ae9f71f1f45003b85eeca2"} Nov 02 20:06:04.532955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.532757 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" event={"ID":"4f1f36d2-2a02-463d-b9ce-73265a4407af","Type":"ContainerDied","Data":"8f0fdd4709a1cee5d1b18838cb11108d050a7fd80005936d5cc0a1c96b79843a"} Nov 02 20:06:04.532955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.532775 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f0fdd4709a1cee5d1b18838cb11108d050a7fd80005936d5cc0a1c96b79843a" Nov 02 20:06:04.552441 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.552411 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 20:06:04.708484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.708403 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1f36d2-2a02-463d-b9ce-73265a4407af-kube-api-access-klkpm\" (UniqueName: \"kubernetes.io/projected/4f1f36d2-2a02-463d-b9ce-73265a4407af-kube-api-access-klkpm\") pod \"4f1f36d2-2a02-463d-b9ce-73265a4407af\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " Nov 02 20:06:04.708484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.708480 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1f36d2-2a02-463d-b9ce-73265a4407af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1f36d2-2a02-463d-b9ce-73265a4407af-plugins-config-volume\") pod \"4f1f36d2-2a02-463d-b9ce-73265a4407af\" (UID: \"4f1f36d2-2a02-463d-b9ce-73265a4407af\") " Nov 02 20:06:04.708882 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.708856 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1f36d2-2a02-463d-b9ce-73265a4407af-plugins-config-volume" pod "4f1f36d2-2a02-463d-b9ce-73265a4407af" (UID: "4f1f36d2-2a02-463d-b9ce-73265a4407af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:04.709169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.709151 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1f36d2-2a02-463d-b9ce-73265a4407af-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:04.710627 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.710599 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1f36d2-2a02-463d-b9ce-73265a4407af-kube-api-access-klkpm" pod "4f1f36d2-2a02-463d-b9ce-73265a4407af" (UID: "4f1f36d2-2a02-463d-b9ce-73265a4407af"). InnerVolumeSpecName "kube-api-access-klkpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:04.810798 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:04.810765 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klkpm\" (UniqueName: \"kubernetes.io/projected/4f1f36d2-2a02-463d-b9ce-73265a4407af-kube-api-access-klkpm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:05.543993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:05.543951 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98" Nov 02 20:06:05.698234 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:05.698200 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98"] Nov 02 20:06:05.747765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:05.747724 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-28623b-566849489c-pnr98"] Nov 02 20:06:06.191141 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:06.191099 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1f36d2-2a02-463d-b9ce-73265a4407af" path="/var/lib/kubelet/pods/4f1f36d2-2a02-463d-b9ce-73265a4407af/volumes" Nov 02 20:06:10.151418 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.151374 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb"] Nov 02 20:06:10.152925 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.152891 4966 cpu_manager.go:414] "containerMap: removing container" podUID="bd2b3ee3-285e-4740-aee2-7003aa50baed" containerName="payload-processing" Nov 02 20:06:10.153109 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153091 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2b3ee3-285e-4740-aee2-7003aa50baed" containerName="payload-processing" Nov 02 20:06:10.153227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153215 4966 cpu_manager.go:414] "containerMap: removing container" podUID="989c00b9-29a1-448d-81cf-7e76c5b254ee" containerName="revoke-keys" Nov 02 20:06:10.153332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153320 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989c00b9-29a1-448d-81cf-7e76c5b254ee" containerName="revoke-keys" Nov 02 20:06:10.153442 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153432 4966 cpu_manager.go:414] "containerMap: removing container" podUID="285f6908-ff18-4259-b37f-b0a4ea9acb64" containerName="revoke-keys" Nov 02 20:06:10.153533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153520 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285f6908-ff18-4259-b37f-b0a4ea9acb64" containerName="revoke-keys" Nov 02 20:06:10.153639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153626 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4f1f36d2-2a02-463d-b9ce-73265a4407af" containerName="payload-processing" Nov 02 20:06:10.153721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153710 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1f36d2-2a02-463d-b9ce-73265a4407af" containerName="payload-processing" Nov 02 20:06:10.153991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.153974 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="285f6908-ff18-4259-b37f-b0a4ea9acb64" containerName="revoke-keys" Nov 02 20:06:10.154103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.154091 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1f36d2-2a02-463d-b9ce-73265a4407af" containerName="payload-processing" Nov 02 20:06:10.154201 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.154190 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2b3ee3-285e-4740-aee2-7003aa50baed" containerName="payload-processing" Nov 02 20:06:10.154281 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.154271 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="989c00b9-29a1-448d-81cf-7e76c5b254ee" containerName="revoke-keys" Nov 02 20:06:10.165906 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.165879 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.202533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.202500 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb"] Nov 02 20:06:10.241521 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.241331 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:06:10.241521 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.241432 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:06:10.241521 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.241491 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:06:10.264073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.264028 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:06:10.297320 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.291127 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.297320 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.291488 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcc8\" (UniqueName: \"kubernetes.io/projected/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-kube-api-access-nfcc8\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.297320 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.291596 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.401549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.401468 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.401549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.401516 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.401775 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.401669 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcc8\" (UniqueName: \"kubernetes.io/projected/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-kube-api-access-nfcc8\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.402361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.402275 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.404258 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.404231 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.455776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.455742 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcc8\" (UniqueName: \"kubernetes.io/projected/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-kube-api-access-nfcc8\") pod \"maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:10.495329 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:10.491908 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:11.030875 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:11.030837 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb"] Nov 02 20:06:11.034697 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:06:11.034665 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1903cc_89cd_4ab9_aed3_a7c22252d8b1.slice/crio-dcb2e3267b4c4e770633c44c5280c5b20ead77b6e2ceb3f8b284f5f1ed7c7a91 WatchSource:0}: Error finding container dcb2e3267b4c4e770633c44c5280c5b20ead77b6e2ceb3f8b284f5f1ed7c7a91: Status 404 returned error can't find the container with id dcb2e3267b4c4e770633c44c5280c5b20ead77b6e2ceb3f8b284f5f1ed7c7a91 Nov 02 20:06:11.644572 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:11.644404 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" event={"ID":"be1903cc-89cd-4ab9-aed3-a7c22252d8b1","Type":"ContainerStarted","Data":"7cf4993d32fd4a0980658c9b4d82ad9b364b6c59c33135c10d74fc59674a1bb5"} Nov 02 20:06:11.644572 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:11.644476 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" event={"ID":"be1903cc-89cd-4ab9-aed3-a7c22252d8b1","Type":"ContainerStarted","Data":"dcb2e3267b4c4e770633c44c5280c5b20ead77b6e2ceb3f8b284f5f1ed7c7a91"} Nov 02 20:06:11.757496 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:11.757433 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" podStartSLOduration=1.7574161830000001 podStartE2EDuration="1.757416183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:11.753677303 +0000 UTC m=+2277.702333142" watchObservedRunningTime="2026-11-02 20:06:11.757416183 +0000 UTC m=+2277.706072051" Nov 02 20:06:12.658086 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:12.658046 4966 generic.go:356] "Generic (PLEG): container finished" podID="be1903cc-89cd-4ab9-aed3-a7c22252d8b1" containerID="7cf4993d32fd4a0980658c9b4d82ad9b364b6c59c33135c10d74fc59674a1bb5" exitCode=0 Nov 02 20:06:12.658644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:12.658108 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" event={"ID":"be1903cc-89cd-4ab9-aed3-a7c22252d8b1","Type":"ContainerDied","Data":"7cf4993d32fd4a0980658c9b4d82ad9b364b6c59c33135c10d74fc59674a1bb5"} Nov 02 20:06:13.990611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:13.990579 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:14.050142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.050102 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-kube-api-access-nfcc8\" (UniqueName: \"kubernetes.io/projected/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-kube-api-access-nfcc8\") pod \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " Nov 02 20:06:14.050370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.050156 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-xks-ca\") pod \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " Nov 02 20:06:14.050370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.050191 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-openshift-ca\") pod \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\" (UID: \"be1903cc-89cd-4ab9-aed3-a7c22252d8b1\") " Nov 02 20:06:14.051934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.051033 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-openshift-ca" pod "be1903cc-89cd-4ab9-aed3-a7c22252d8b1" (UID: "be1903cc-89cd-4ab9-aed3-a7c22252d8b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:14.053053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.053025 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-xks-ca" pod "be1903cc-89cd-4ab9-aed3-a7c22252d8b1" (UID: "be1903cc-89cd-4ab9-aed3-a7c22252d8b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:06:14.053903 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.053873 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-kube-api-access-nfcc8" pod "be1903cc-89cd-4ab9-aed3-a7c22252d8b1" (UID: "be1903cc-89cd-4ab9-aed3-a7c22252d8b1"). InnerVolumeSpecName "kube-api-access-nfcc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:14.155106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.152215 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfcc8\" (UniqueName: \"kubernetes.io/projected/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-kube-api-access-nfcc8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:14.155106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.152254 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:14.155106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.152268 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1903cc-89cd-4ab9-aed3-a7c22252d8b1-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:14.694928 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.694887 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" event={"ID":"be1903cc-89cd-4ab9-aed3-a7c22252d8b1","Type":"ContainerDied","Data":"dcb2e3267b4c4e770633c44c5280c5b20ead77b6e2ceb3f8b284f5f1ed7c7a91"} Nov 02 20:06:14.695189 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.695166 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb2e3267b4c4e770633c44c5280c5b20ead77b6e2ceb3f8b284f5f1ed7c7a91" Nov 02 20:06:14.695437 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.695419 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb" Nov 02 20:06:14.700755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:14.700728 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9"] Nov 02 20:06:14.762254 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:06:14.762217 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:06:18.502456 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:18.502420 4966 scope.go:122] "RemoveContainer" containerID="70aeeb92c6761c5a3a373938fc89490e4db0549e26ae9f71f1f45003b85eeca2" Nov 02 20:06:29.724086 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:29.723976 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" containerID="cri-o://ab29deba28ccbf6418a0a8344f8fe72a47f5309798562c3e5dcd46e2bedf9a65" gracePeriod=45 Nov 02 20:06:29.848739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:29.848704 4966 generic.go:356] "Generic (PLEG): container finished" podID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerID="ab29deba28ccbf6418a0a8344f8fe72a47f5309798562c3e5dcd46e2bedf9a65" exitCode=0 Nov 02 20:06:29.848875 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:29.848781 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" event={"ID":"519e8de1-8721-4152-b336-f9cfcf3ab87c","Type":"ContainerDied","Data":"ab29deba28ccbf6418a0a8344f8fe72a47f5309798562c3e5dcd46e2bedf9a65"} Nov 02 20:06:30.005494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.005469 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 20:06:30.084402 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.084367 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519e8de1-8721-4152-b336-f9cfcf3ab87c-kube-api-access-gtssc\" (UniqueName: \"kubernetes.io/projected/519e8de1-8721-4152-b336-f9cfcf3ab87c-kube-api-access-gtssc\") pod \"519e8de1-8721-4152-b336-f9cfcf3ab87c\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " Nov 02 20:06:30.084678 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.084649 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-model-cache\") pod \"519e8de1-8721-4152-b336-f9cfcf3ab87c\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " Nov 02 20:06:30.084809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.084793 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-dshm\") pod \"519e8de1-8721-4152-b336-f9cfcf3ab87c\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " Nov 02 20:06:30.084860 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.084812 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-home\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-home\") pod \"519e8de1-8721-4152-b336-f9cfcf3ab87c\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " Nov 02 20:06:30.084924 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.084870 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-tmp-dir\") pod \"519e8de1-8721-4152-b336-f9cfcf3ab87c\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " Nov 02 20:06:30.084924 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.084891 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519e8de1-8721-4152-b336-f9cfcf3ab87c-tls-certs\" (UniqueName: \"kubernetes.io/secret/519e8de1-8721-4152-b336-f9cfcf3ab87c-tls-certs\") pod \"519e8de1-8721-4152-b336-f9cfcf3ab87c\" (UID: \"519e8de1-8721-4152-b336-f9cfcf3ab87c\") " Nov 02 20:06:30.092382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.087091 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-home" pod "519e8de1-8721-4152-b336-f9cfcf3ab87c" (UID: "519e8de1-8721-4152-b336-f9cfcf3ab87c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.092382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.087186 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-model-cache" pod "519e8de1-8721-4152-b336-f9cfcf3ab87c" (UID: "519e8de1-8721-4152-b336-f9cfcf3ab87c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.092382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.087385 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-tmp-dir" pod "519e8de1-8721-4152-b336-f9cfcf3ab87c" (UID: "519e8de1-8721-4152-b336-f9cfcf3ab87c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.092382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.090008 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519e8de1-8721-4152-b336-f9cfcf3ab87c-kube-api-access-gtssc" pod "519e8de1-8721-4152-b336-f9cfcf3ab87c" (UID: "519e8de1-8721-4152-b336-f9cfcf3ab87c"). InnerVolumeSpecName "kube-api-access-gtssc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:30.092382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.090705 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519e8de1-8721-4152-b336-f9cfcf3ab87c-tls-certs" pod "519e8de1-8721-4152-b336-f9cfcf3ab87c" (UID: "519e8de1-8721-4152-b336-f9cfcf3ab87c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:06:30.093083 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.093058 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-dshm" pod "519e8de1-8721-4152-b336-f9cfcf3ab87c" (UID: "519e8de1-8721-4152-b336-f9cfcf3ab87c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:06:30.187528 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.187485 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.187528 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.187531 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.187793 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.187548 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.187793 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.187562 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/519e8de1-8721-4152-b336-f9cfcf3ab87c-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.187793 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.187575 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/519e8de1-8721-4152-b336-f9cfcf3ab87c-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.187793 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.187593 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtssc\" (UniqueName: \"kubernetes.io/projected/519e8de1-8721-4152-b336-f9cfcf3ab87c-kube-api-access-gtssc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:30.871352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.870951 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" event={"ID":"519e8de1-8721-4152-b336-f9cfcf3ab87c","Type":"ContainerDied","Data":"d7217bca892046f86d858ea1bf7055f9be9766c40ae49507163e409857844ea9"} Nov 02 20:06:30.871352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.871028 4966 scope.go:122] "RemoveContainer" containerID="ab29deba28ccbf6418a0a8344f8fe72a47f5309798562c3e5dcd46e2bedf9a65" Nov 02 20:06:30.871352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:30.871252 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9" Nov 02 20:06:31.030081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:31.030045 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9"] Nov 02 20:06:31.036106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:31.036069 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8795bf/simulator-w4-8795bf-kserve-5f4b856cd6-x8pd9"] Nov 02 20:06:32.187013 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:32.186978 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" path="/var/lib/kubelet/pods/519e8de1-8721-4152-b336-f9cfcf3ab87c/volumes" Nov 02 20:06:33.618842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.618806 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs"] Nov 02 20:06:33.619425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.619403 4966 cpu_manager.go:414] "containerMap: removing container" podUID="be1903cc-89cd-4ab9-aed3-a7c22252d8b1" containerName="revoke-keys" Nov 02 20:06:33.619425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.619419 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1903cc-89cd-4ab9-aed3-a7c22252d8b1" containerName="revoke-keys" Nov 02 20:06:33.619549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.619478 4966 cpu_manager.go:414] "containerMap: removing container" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" Nov 02 20:06:33.619549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.619484 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" Nov 02 20:06:33.619621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.619581 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="519e8de1-8721-4152-b336-f9cfcf3ab87c" containerName="main" Nov 02 20:06:33.619621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.619594 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1903cc-89cd-4ab9-aed3-a7c22252d8b1" containerName="revoke-keys" Nov 02 20:06:33.624705 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.624685 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:33.678033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.677993 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs"] Nov 02 20:06:33.717862 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.717814 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-13e739\"" Nov 02 20:06:33.718044 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.717865 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-13e739-dockercfg-lpmb9\"" Nov 02 20:06:33.762458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.762389 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8754ec15-5b0b-4188-9f35-4d00a8b24043-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:33.762730 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.762712 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jv9\" (UniqueName: \"kubernetes.io/projected/8754ec15-5b0b-4188-9f35-4d00a8b24043-kube-api-access-x2jv9\") pod \"payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:33.863794 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.863746 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8754ec15-5b0b-4188-9f35-4d00a8b24043-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:33.863990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.863901 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jv9\" (UniqueName: \"kubernetes.io/projected/8754ec15-5b0b-4188-9f35-4d00a8b24043-kube-api-access-x2jv9\") pod \"payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:33.864845 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.864815 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8754ec15-5b0b-4188-9f35-4d00a8b24043-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:33.916988 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.916913 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jv9\" (UniqueName: \"kubernetes.io/projected/8754ec15-5b0b-4188-9f35-4d00a8b24043-kube-api-access-x2jv9\") pod \"payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:33.943877 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:33.943828 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:34.321377 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:06:34.321345 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8754ec15_5b0b_4188_9f35_4d00a8b24043.slice/crio-b1e71fdf39fff96ef7129ab58f4e37c3903ade85b0e44d9da05661ef9b013880 WatchSource:0}: Error finding container b1e71fdf39fff96ef7129ab58f4e37c3903ade85b0e44d9da05661ef9b013880: Status 404 returned error can't find the container with id b1e71fdf39fff96ef7129ab58f4e37c3903ade85b0e44d9da05661ef9b013880 Nov 02 20:06:34.322154 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:34.322125 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs"] Nov 02 20:06:34.908536 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:34.908495 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" event={"ID":"8754ec15-5b0b-4188-9f35-4d00a8b24043","Type":"ContainerStarted","Data":"57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc"} Nov 02 20:06:34.908955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:34.908566 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:06:34.908955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:34.908594 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" event={"ID":"8754ec15-5b0b-4188-9f35-4d00a8b24043","Type":"ContainerStarted","Data":"b1e71fdf39fff96ef7129ab58f4e37c3903ade85b0e44d9da05661ef9b013880"} Nov 02 20:06:35.016740 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:35.016673 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" podStartSLOduration=2.016656796 podStartE2EDuration="2.016656796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:35.013546402 +0000 UTC m=+2300.962202241" watchObservedRunningTime="2026-11-02 20:06:35.016656796 +0000 UTC m=+2300.965312634" Nov 02 20:06:44.754551 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:06:44.754512 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:06:45.919272 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:06:45.919242 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:07:03.355641 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:03.355598 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2"] Nov 02 20:07:03.397313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:03.397269 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-28623b-wlcg2"] Nov 02 20:07:04.186502 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:04.186466 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19992d69-4c32-4e6e-8499-4f5738f6bd0e" path="/var/lib/kubelet/pods/19992d69-4c32-4e6e-8499-4f5738f6bd0e/volumes" Nov 02 20:07:09.174550 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:09.174509 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh"] Nov 02 20:07:09.219548 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:09.219512 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1d869e-kqsvh"] Nov 02 20:07:09.903005 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:09.902974 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r"] Nov 02 20:07:09.903205 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:09.903177 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" containerID="cri-o://a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa" gracePeriod=30 Nov 02 20:07:09.915489 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:09.915457 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft"] Nov 02 20:07:09.915654 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:09.915635 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" containerID="cri-o://dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89" gracePeriod=30 Nov 02 20:07:10.197500 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:10.197407 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800d29f5-0c7d-4b35-9902-bc70a2df8cc7" path="/var/lib/kubelet/pods/800d29f5-0c7d-4b35-9902-bc70a2df8cc7/volumes" Nov 02 20:07:11.620556 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:11.620515 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:07:11.620971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:11.620584 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:07:13.546544 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:13.546507 4966 patch_prober.go:29] interesting pod/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:07:13.547053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:13.546577 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:07:14.761075 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:07:14.761031 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:07:15.213396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.213357 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 20:07:15.260867 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.260837 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 20:07:15.274676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.274575 4966 generic.go:356] "Generic (PLEG): container finished" podID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerID="a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa" exitCode=0 Nov 02 20:07:15.274676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.274626 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" Nov 02 20:07:15.274676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.274651 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" event={"ID":"c0861dab-4be6-4d9e-b2bb-e7f214200037","Type":"ContainerDied","Data":"a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa"} Nov 02 20:07:15.274979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.274715 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r" event={"ID":"c0861dab-4be6-4d9e-b2bb-e7f214200037","Type":"ContainerDied","Data":"77bf809e28bf02918d4115ff17f7c7619fc6133eeee7e0c8e00671ddd463ad48"} Nov 02 20:07:15.274979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.274747 4966 scope.go:122] "RemoveContainer" containerID="a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa" Nov 02 20:07:15.276659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.276637 4966 generic.go:356] "Generic (PLEG): container finished" podID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerID="dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89" exitCode=0 Nov 02 20:07:15.276786 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.276706 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" Nov 02 20:07:15.276786 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.276712 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" event={"ID":"acbf3a24-63a8-4e99-898d-4a95efa38833","Type":"ContainerDied","Data":"dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89"} Nov 02 20:07:15.276786 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.276756 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft" event={"ID":"acbf3a24-63a8-4e99-898d-4a95efa38833","Type":"ContainerDied","Data":"0772c9939c440a5f6345b4e892eac61ab9371e0c34ece84af7c465d787a56e0a"} Nov 02 20:07:15.287862 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.287832 4966 scope.go:122] "RemoveContainer" containerID="618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b" Nov 02 20:07:15.298423 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.298401 4966 scope.go:122] "RemoveContainer" containerID="a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa" Nov 02 20:07:15.298782 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:07:15.298762 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa\": container with ID starting with a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa not found: ID does not exist" containerID="a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa" Nov 02 20:07:15.298833 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.298793 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa"} err="failed to get container status \"a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa\": rpc error: code = NotFound desc = could not find container \"a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa\": container with ID starting with a89556edb1f9eeaa63125714d515db20eb5cd2b46816ba671a2e82258c03a0aa not found: ID does not exist" Nov 02 20:07:15.298833 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.298814 4966 scope.go:122] "RemoveContainer" containerID="618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b" Nov 02 20:07:15.299098 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:07:15.299069 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b\": container with ID starting with 618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b not found: ID does not exist" containerID="618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b" Nov 02 20:07:15.299178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.299105 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b"} err="failed to get container status \"618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b\": rpc error: code = NotFound desc = could not find container \"618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b\": container with ID starting with 618a3df074c414a7cb8da2c0facb0dbf15ebcb54c6f08ee3843e87e6303fee7b not found: ID does not exist" Nov 02 20:07:15.299178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.299131 4966 scope.go:122] "RemoveContainer" containerID="dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89" Nov 02 20:07:15.300533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300516 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-envoy\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300594 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0861dab-4be6-4d9e-b2bb-e7f214200037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0861dab-4be6-4d9e-b2bb-e7f214200037-istiod-ca-cert\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300651 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300622 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-data\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300657 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-token\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-token\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300729 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300715 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-kube-api-access-fp6zv\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-kube-api-access-fp6zv\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300768 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300738 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-certs\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300770 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-socket\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300799 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-podinfo\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.300922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300887 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-credential-socket\") pod \"c0861dab-4be6-4d9e-b2bb-e7f214200037\" (UID: \"c0861dab-4be6-4d9e-b2bb-e7f214200037\") " Nov 02 20:07:15.301010 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.300989 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0861dab-4be6-4d9e-b2bb-e7f214200037-istiod-ca-cert" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:07:15.301077 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.301041 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-certs" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.301261 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.301237 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-socket" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.301511 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.301427 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-credential-socket" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.301817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.301772 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0861dab-4be6-4d9e-b2bb-e7f214200037-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.301995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.301822 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.301995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.301838 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.301995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.301851 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.302185 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.302141 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-data" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.303202 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.303164 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-envoy" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.303451 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.303421 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-token" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:15.303565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.303479 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-kube-api-access-fp6zv" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "kube-api-access-fp6zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:15.303565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.303504 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-podinfo" pod "c0861dab-4be6-4d9e-b2bb-e7f214200037" (UID: "c0861dab-4be6-4d9e-b2bb-e7f214200037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:07:15.311181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.311158 4966 scope.go:122] "RemoveContainer" containerID="dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89" Nov 02 20:07:15.311530 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:07:15.311507 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89\": container with ID starting with dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89 not found: ID does not exist" containerID="dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89" Nov 02 20:07:15.311598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.311542 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89"} err="failed to get container status \"dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89\": rpc error: code = NotFound desc = could not find container \"dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89\": container with ID starting with dd03002ec49ef15082e5c6f79e2b6e6b87715f7c7a93d108ee93cf8127858c89 not found: ID does not exist" Nov 02 20:07:15.402492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402455 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-certs\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.402692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402543 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-socket\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.402692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402575 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbf3a24-63a8-4e99-898d-4a95efa38833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbf3a24-63a8-4e99-898d-4a95efa38833-istiod-ca-cert\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.402692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402652 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-envoy\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.402871 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402696 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-data\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.402871 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402750 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-kube-api-access-bhl6p\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-kube-api-access-bhl6p\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.402871 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402763 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-certs" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.402871 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402799 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-credential-socket\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.403080 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402872 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-podinfo\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.403080 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403012 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-token\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-token\") pod \"acbf3a24-63a8-4e99-898d-4a95efa38833\" (UID: \"acbf3a24-63a8-4e99-898d-4a95efa38833\") " Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403542 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403568 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403582 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403597 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp6zv\" (UniqueName: \"kubernetes.io/projected/c0861dab-4be6-4d9e-b2bb-e7f214200037-kube-api-access-fp6zv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403611 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0861dab-4be6-4d9e-b2bb-e7f214200037-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403624 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.402870 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-socket" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.403713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.403442 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-data" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.404382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.404355 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-credential-socket" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.404545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.404524 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbf3a24-63a8-4e99-898d-4a95efa38833-istiod-ca-cert" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:07:15.405584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.405561 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-envoy" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:15.405685 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.405653 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-kube-api-access-bhl6p" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "kube-api-access-bhl6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:15.406706 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.406674 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-podinfo" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:07:15.406908 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.406888 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-token" pod "acbf3a24-63a8-4e99-898d-4a95efa38833" (UID: "acbf3a24-63a8-4e99-898d-4a95efa38833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504548 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504587 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbf3a24-63a8-4e99-898d-4a95efa38833-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504599 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504613 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504627 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhl6p\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-kube-api-access-bhl6p\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504640 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbf3a24-63a8-4e99-898d-4a95efa38833-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504652 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.504680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.504664 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbf3a24-63a8-4e99-898d-4a95efa38833-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:15.718635 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.718599 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft"] Nov 02 20:07:15.724056 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.724026 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-vbtft"] Nov 02 20:07:15.812878 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.812840 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd"] Nov 02 20:07:15.813398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.813075 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" containerID="cri-o://faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578" gracePeriod=30 Nov 02 20:07:15.832730 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.832702 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r"] Nov 02 20:07:15.851366 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:15.839099 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-28623b-openshift-default-5b66f6f775-ktc6r"] Nov 02 20:07:16.191316 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:16.191262 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" path="/var/lib/kubelet/pods/acbf3a24-63a8-4e99-898d-4a95efa38833/volumes" Nov 02 20:07:16.192098 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:16.192072 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" path="/var/lib/kubelet/pods/c0861dab-4be6-4d9e-b2bb-e7f214200037/volumes" Nov 02 20:07:17.667279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.667239 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24"] Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.667855 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.667872 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.667883 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.667888 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.667912 4966 cpu_manager.go:414] "containerMap: removing container" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.667917 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.668005 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.668012 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0861dab-4be6-4d9e-b2bb-e7f214200037" containerName="istio-proxy" Nov 02 20:07:17.668091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.668022 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbf3a24-63a8-4e99-898d-4a95efa38833" containerName="istio-proxy" Nov 02 20:07:17.673342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.673322 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.718075 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.718034 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk"] Nov 02 20:07:17.722385 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.722362 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.748925 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.748880 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.749136 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.748943 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcrw\" (UniqueName: \"kubernetes.io/projected/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-kube-api-access-kvcrw\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.749136 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.749106 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.760066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.760030 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:07:17.764870 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.764838 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24"] Nov 02 20:07:17.768406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.768131 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:07:17.768406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.768235 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:07:17.768406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.768367 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:07:17.783807 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.783770 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk"] Nov 02 20:07:17.851114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.851056 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.851376 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.851154 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88992671-b828-4f6d-818f-16cc845d843c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.851376 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.851183 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcrw\" (UniqueName: \"kubernetes.io/projected/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-kube-api-access-kvcrw\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.851376 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.851348 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88992671-b828-4f6d-818f-16cc845d843c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.851574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.851465 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.851574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.851503 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sl7\" (UniqueName: \"kubernetes.io/projected/88992671-b828-4f6d-818f-16cc845d843c-kube-api-access-q2sl7\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.852382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.852355 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.853984 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.853962 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.903880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.903841 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcrw\" (UniqueName: \"kubernetes.io/projected/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-kube-api-access-kvcrw\") pod \"maas-api-revoke-sub-fc9a48a3-jjl24\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:17.952658 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.952573 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88992671-b828-4f6d-818f-16cc845d843c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.952815 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.952723 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88992671-b828-4f6d-818f-16cc845d843c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.952815 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.952779 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sl7\" (UniqueName: \"kubernetes.io/projected/88992671-b828-4f6d-818f-16cc845d843c-kube-api-access-q2sl7\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.953616 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.953582 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88992671-b828-4f6d-818f-16cc845d843c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.955188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.955164 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88992671-b828-4f6d-818f-16cc845d843c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:17.988171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:17.988140 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:18.005180 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:18.005153 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sl7\" (UniqueName: \"kubernetes.io/projected/88992671-b828-4f6d-818f-16cc845d843c-kube-api-access-q2sl7\") pod \"maas-api-revoke-sub-49806dc8-9vzgk\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:18.039047 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:18.039008 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:18.368053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:18.368013 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24"] Nov 02 20:07:18.373004 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:07:18.372973 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554e692d_c8a4_4b6b_bef1_2e2a4313c9ec.slice/crio-a191783ddb78693737a6d6b62d75c4ea74875c8782804a0c1ce6399e42872540 WatchSource:0}: Error finding container a191783ddb78693737a6d6b62d75c4ea74875c8782804a0c1ce6399e42872540: Status 404 returned error can't find the container with id a191783ddb78693737a6d6b62d75c4ea74875c8782804a0c1ce6399e42872540 Nov 02 20:07:18.477193 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:18.477157 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk"] Nov 02 20:07:18.540869 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:07:18.540832 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88992671_b828_4f6d_818f_16cc845d843c.slice/crio-64a88f7032a60044d008a1736e23dc35ced700a0a9c7649278e9d7a2abdccd17 WatchSource:0}: Error finding container 64a88f7032a60044d008a1736e23dc35ced700a0a9c7649278e9d7a2abdccd17: Status 404 returned error can't find the container with id 64a88f7032a60044d008a1736e23dc35ced700a0a9c7649278e9d7a2abdccd17 Nov 02 20:07:18.551903 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:18.550597 4966 scope.go:122] "RemoveContainer" containerID="dd607ea0c4a36d5d68444efb5b5089a685c8a9c2cc87e229cd62010b5e750df9" Nov 02 20:07:19.384830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:19.384791 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" event={"ID":"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec","Type":"ContainerStarted","Data":"3c3b1e6da060885fcea4674ad29b0fc7a51e684503a7b2dde87ebb6d2b94cbba"} Nov 02 20:07:19.398149 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:19.390851 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" event={"ID":"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec","Type":"ContainerStarted","Data":"a191783ddb78693737a6d6b62d75c4ea74875c8782804a0c1ce6399e42872540"} Nov 02 20:07:19.400845 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:19.400787 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" event={"ID":"88992671-b828-4f6d-818f-16cc845d843c","Type":"ContainerStarted","Data":"a5b781d29a8716d847842b2fcd9ec369970ac5a66628511498b9f8c787fb28f5"} Nov 02 20:07:19.400988 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:19.400883 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" event={"ID":"88992671-b828-4f6d-818f-16cc845d843c","Type":"ContainerStarted","Data":"64a88f7032a60044d008a1736e23dc35ced700a0a9c7649278e9d7a2abdccd17"} Nov 02 20:07:19.499178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:19.499124 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" podStartSLOduration=2.499110002 podStartE2EDuration="2.499110002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:19.496590698 +0000 UTC m=+2345.445246539" watchObservedRunningTime="2026-11-02 20:07:19.499110002 +0000 UTC m=+2345.447765841" Nov 02 20:07:19.603664 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:19.603588 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" podStartSLOduration=2.603570657 podStartE2EDuration="2.603570657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:19.601477354 +0000 UTC m=+2345.550133192" watchObservedRunningTime="2026-11-02 20:07:19.603570657 +0000 UTC m=+2345.552226498" Nov 02 20:07:20.466795 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:20.466750 4966 generic.go:356] "Generic (PLEG): container finished" podID="554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" containerID="3c3b1e6da060885fcea4674ad29b0fc7a51e684503a7b2dde87ebb6d2b94cbba" exitCode=0 Nov 02 20:07:20.484993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:20.471399 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" event={"ID":"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec","Type":"ContainerDied","Data":"3c3b1e6da060885fcea4674ad29b0fc7a51e684503a7b2dde87ebb6d2b94cbba"} Nov 02 20:07:20.485422 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:20.484015 4966 generic.go:356] "Generic (PLEG): container finished" podID="88992671-b828-4f6d-818f-16cc845d843c" containerID="a5b781d29a8716d847842b2fcd9ec369970ac5a66628511498b9f8c787fb28f5" exitCode=0 Nov 02 20:07:20.485422 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:20.485372 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" event={"ID":"88992671-b828-4f6d-818f-16cc845d843c","Type":"ContainerDied","Data":"a5b781d29a8716d847842b2fcd9ec369970ac5a66628511498b9f8c787fb28f5"} Nov 02 20:07:21.116840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.116812 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 20:07:21.236089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236044 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4550e9d0-4fb2-451e-a67d-5996dae13934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550e9d0-4fb2-451e-a67d-5996dae13934-istiod-ca-cert\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236120 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-data\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236168 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-socket\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236262 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-podinfo\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236287 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-kube-api-access-zg8ss\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-kube-api-access-zg8ss\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236566 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236526 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-envoy\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236660 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-token\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-token\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236708 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-certs\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.236749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.236739 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-credential-socket\") pod \"4550e9d0-4fb2-451e-a67d-5996dae13934\" (UID: \"4550e9d0-4fb2-451e-a67d-5996dae13934\") " Nov 02 20:07:21.239122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.238439 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-socket" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:21.239122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.238515 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-credential-socket" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:21.239122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.238699 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-certs" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:21.239122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.239082 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4550e9d0-4fb2-451e-a67d-5996dae13934-istiod-ca-cert" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:07:21.239753 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.239714 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-data" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:21.246462 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.246424 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-podinfo" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:07:21.246810 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.246766 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-token" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:21.247239 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.247201 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-kube-api-access-zg8ss" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "kube-api-access-zg8ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:21.248016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.247956 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-envoy" pod "4550e9d0-4fb2-451e-a67d-5996dae13934" (UID: "4550e9d0-4fb2-451e-a67d-5996dae13934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338709 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4550e9d0-4fb2-451e-a67d-5996dae13934-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338745 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338756 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338764 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338773 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg8ss\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-kube-api-access-zg8ss\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338784 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338792 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4550e9d0-4fb2-451e-a67d-5996dae13934-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338800 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.338801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.338808 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4550e9d0-4fb2-451e-a67d-5996dae13934-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.496463 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.496422 4966 generic.go:356] "Generic (PLEG): container finished" podID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerID="faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578" exitCode=0 Nov 02 20:07:21.496922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.496475 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" Nov 02 20:07:21.496922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.496515 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" event={"ID":"4550e9d0-4fb2-451e-a67d-5996dae13934","Type":"ContainerDied","Data":"faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578"} Nov 02 20:07:21.496922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.496571 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd" event={"ID":"4550e9d0-4fb2-451e-a67d-5996dae13934","Type":"ContainerDied","Data":"7a9940cd64e381dcfad26f70b509667d0f3ffd64a230289fc8a0b533dafb29ed"} Nov 02 20:07:21.496922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.496593 4966 scope.go:122] "RemoveContainer" containerID="faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578" Nov 02 20:07:21.509732 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.509709 4966 scope.go:122] "RemoveContainer" containerID="faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578" Nov 02 20:07:21.510132 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:07:21.510103 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578\": container with ID starting with faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578 not found: ID does not exist" containerID="faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578" Nov 02 20:07:21.510235 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.510146 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578"} err="failed to get container status \"faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578\": rpc error: code = NotFound desc = could not find container \"faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578\": container with ID starting with faf27ee9ea5e046ad2e7199970207876985b5bfa4828afe7c811da398ed24578 not found: ID does not exist" Nov 02 20:07:21.669038 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.668940 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd"] Nov 02 20:07:21.696391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.696291 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:21.699680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.699654 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:21.705795 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.705752 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1d869e-openshift-default-56b959b4df-z75pd"] Nov 02 20:07:21.845668 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.845634 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-kube-api-access-kvcrw\" (UniqueName: \"kubernetes.io/projected/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-kube-api-access-kvcrw\") pod \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " Nov 02 20:07:21.845820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.845687 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88992671-b828-4f6d-818f-16cc845d843c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88992671-b828-4f6d-818f-16cc845d843c-maas-api-openshift-ca\") pod \"88992671-b828-4f6d-818f-16cc845d843c\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " Nov 02 20:07:21.845820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.845728 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-xks-ca\") pod \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " Nov 02 20:07:21.845940 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.845834 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-openshift-ca\") pod \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\" (UID: \"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec\") " Nov 02 20:07:21.845940 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.845902 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88992671-b828-4f6d-818f-16cc845d843c-kube-api-access-q2sl7\" (UniqueName: \"kubernetes.io/projected/88992671-b828-4f6d-818f-16cc845d843c-kube-api-access-q2sl7\") pod \"88992671-b828-4f6d-818f-16cc845d843c\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " Nov 02 20:07:21.846047 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.845973 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88992671-b828-4f6d-818f-16cc845d843c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88992671-b828-4f6d-818f-16cc845d843c-maas-api-xks-ca\") pod \"88992671-b828-4f6d-818f-16cc845d843c\" (UID: \"88992671-b828-4f6d-818f-16cc845d843c\") " Nov 02 20:07:21.846243 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.846216 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-openshift-ca" pod "554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" (UID: "554e692d-c8a4-4b6b-bef1-2e2a4313c9ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:07:21.846483 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.846455 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88992671-b828-4f6d-818f-16cc845d843c-maas-api-openshift-ca" pod "88992671-b828-4f6d-818f-16cc845d843c" (UID: "88992671-b828-4f6d-818f-16cc845d843c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:07:21.846732 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.846708 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88992671-b828-4f6d-818f-16cc845d843c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.846816 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.846737 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.848790 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.848750 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88992671-b828-4f6d-818f-16cc845d843c-kube-api-access-q2sl7" pod "88992671-b828-4f6d-818f-16cc845d843c" (UID: "88992671-b828-4f6d-818f-16cc845d843c"). InnerVolumeSpecName "kube-api-access-q2sl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:21.848901 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.848837 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-kube-api-access-kvcrw" pod "554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" (UID: "554e692d-c8a4-4b6b-bef1-2e2a4313c9ec"). InnerVolumeSpecName "kube-api-access-kvcrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:07:21.849061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.849037 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-xks-ca" pod "554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" (UID: "554e692d-c8a4-4b6b-bef1-2e2a4313c9ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:07:21.849244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.849220 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88992671-b828-4f6d-818f-16cc845d843c-maas-api-xks-ca" pod "88992671-b828-4f6d-818f-16cc845d843c" (UID: "88992671-b828-4f6d-818f-16cc845d843c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:07:21.947932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.947839 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcrw\" (UniqueName: \"kubernetes.io/projected/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-kube-api-access-kvcrw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.947932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.947874 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.947932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.947884 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2sl7\" (UniqueName: \"kubernetes.io/projected/88992671-b828-4f6d-818f-16cc845d843c-kube-api-access-q2sl7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:21.947932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:21.947893 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88992671-b828-4f6d-818f-16cc845d843c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:07:22.186693 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:22.186655 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" path="/var/lib/kubelet/pods/4550e9d0-4fb2-451e-a67d-5996dae13934/volumes" Nov 02 20:07:22.507599 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:22.507567 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" Nov 02 20:07:22.507599 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:22.507584 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk" event={"ID":"88992671-b828-4f6d-818f-16cc845d843c","Type":"ContainerDied","Data":"64a88f7032a60044d008a1736e23dc35ced700a0a9c7649278e9d7a2abdccd17"} Nov 02 20:07:22.508150 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:22.507632 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a88f7032a60044d008a1736e23dc35ced700a0a9c7649278e9d7a2abdccd17" Nov 02 20:07:22.509259 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:22.509238 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" Nov 02 20:07:22.509259 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:22.509244 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24" event={"ID":"554e692d-c8a4-4b6b-bef1-2e2a4313c9ec","Type":"ContainerDied","Data":"a191783ddb78693737a6d6b62d75c4ea74875c8782804a0c1ce6399e42872540"} Nov 02 20:07:22.509468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:07:22.509290 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a191783ddb78693737a6d6b62d75c4ea74875c8782804a0c1ce6399e42872540" Nov 02 20:07:44.756798 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:07:44.756730 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:08:01.774931 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.774837 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd"] Nov 02 20:08:01.775726 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.775701 4966 cpu_manager.go:414] "containerMap: removing container" podUID="554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" containerName="revoke-keys" Nov 02 20:08:01.775726 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.775727 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" containerName="revoke-keys" Nov 02 20:08:01.775893 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.775773 4966 cpu_manager.go:414] "containerMap: removing container" podUID="88992671-b828-4f6d-818f-16cc845d843c" containerName="revoke-keys" Nov 02 20:08:01.775893 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.775783 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88992671-b828-4f6d-818f-16cc845d843c" containerName="revoke-keys" Nov 02 20:08:01.775893 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.775808 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" Nov 02 20:08:01.775893 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.775815 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" Nov 02 20:08:01.776068 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.775982 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="88992671-b828-4f6d-818f-16cc845d843c" containerName="revoke-keys" Nov 02 20:08:01.776068 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.776002 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4550e9d0-4fb2-451e-a67d-5996dae13934" containerName="istio-proxy" Nov 02 20:08:01.776068 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.776018 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" containerName="revoke-keys" Nov 02 20:08:01.780724 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.780696 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:01.834332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.833218 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd"] Nov 02 20:08:01.871338 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.868858 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphtr\" (UniqueName: \"kubernetes.io/projected/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-kube-api-access-pphtr\") pod \"payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:01.871338 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.868994 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:01.879321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.873616 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b45a2b-dockercfg-p9cb5\"" Nov 02 20:08:01.879321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.873697 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b45a2b\"" Nov 02 20:08:01.957337 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.957284 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf"] Nov 02 20:08:01.962093 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.962066 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:01.970448 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.970417 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:01.970604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.970524 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphtr\" (UniqueName: \"kubernetes.io/projected/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-kube-api-access-pphtr\") pod \"payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:01.971327 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:01.971269 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:02.015433 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.015402 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf"] Nov 02 20:08:02.023842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.023814 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphtr\" (UniqueName: \"kubernetes.io/projected/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-kube-api-access-pphtr\") pod \"payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:02.072021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.071926 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eabeab-8fb3-40db-9cfc-ae1512113297-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:02.072209 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.072096 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bll\" (UniqueName: \"kubernetes.io/projected/19eabeab-8fb3-40db-9cfc-ae1512113297-kube-api-access-g9bll\") pod \"payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:02.110991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.110952 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:02.175070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.173489 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eabeab-8fb3-40db-9cfc-ae1512113297-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:02.175070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.173633 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bll\" (UniqueName: \"kubernetes.io/projected/19eabeab-8fb3-40db-9cfc-ae1512113297-kube-api-access-g9bll\") pod \"payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:02.175070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.175016 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eabeab-8fb3-40db-9cfc-ae1512113297-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:02.227834 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.227796 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bll\" (UniqueName: \"kubernetes.io/projected/19eabeab-8fb3-40db-9cfc-ae1512113297-kube-api-access-g9bll\") pod \"payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:02.279345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.279308 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:02.489224 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.489194 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd"] Nov 02 20:08:02.490011 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:08:02.489979 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1992fc16_5363_4bc9_a2a9_b6da4d39d63d.slice/crio-682e437efa06eaf3fb34be2bdb8995fa9bd4717fb6cf7bd25447cf5157b3829c WatchSource:0}: Error finding container 682e437efa06eaf3fb34be2bdb8995fa9bd4717fb6cf7bd25447cf5157b3829c: Status 404 returned error can't find the container with id 682e437efa06eaf3fb34be2bdb8995fa9bd4717fb6cf7bd25447cf5157b3829c Nov 02 20:08:02.655600 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.655570 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf"] Nov 02 20:08:02.656505 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:08:02.656477 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19eabeab_8fb3_40db_9cfc_ae1512113297.slice/crio-896003a7a06fc26697132781ea4d7e2dec9e8a105256669dd3af343c6ac49204 WatchSource:0}: Error finding container 896003a7a06fc26697132781ea4d7e2dec9e8a105256669dd3af343c6ac49204: Status 404 returned error can't find the container with id 896003a7a06fc26697132781ea4d7e2dec9e8a105256669dd3af343c6ac49204 Nov 02 20:08:02.919371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.919263 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" event={"ID":"1992fc16-5363-4bc9-a2a9-b6da4d39d63d","Type":"ContainerStarted","Data":"38d8cfd5a6133bb754bf9391352584c45dcaaf90a60ce5559977cd4a56fc91eb"} Nov 02 20:08:02.919371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.919345 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:02.919371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.919369 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" event={"ID":"1992fc16-5363-4bc9-a2a9-b6da4d39d63d","Type":"ContainerStarted","Data":"682e437efa06eaf3fb34be2bdb8995fa9bd4717fb6cf7bd25447cf5157b3829c"} Nov 02 20:08:02.921108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.921077 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" event={"ID":"19eabeab-8fb3-40db-9cfc-ae1512113297","Type":"ContainerStarted","Data":"61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a"} Nov 02 20:08:02.921242 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.921124 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" event={"ID":"19eabeab-8fb3-40db-9cfc-ae1512113297","Type":"ContainerStarted","Data":"896003a7a06fc26697132781ea4d7e2dec9e8a105256669dd3af343c6ac49204"} Nov 02 20:08:02.921242 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:02.921149 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:03.027553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:03.027481 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" podStartSLOduration=2.027462883 podStartE2EDuration="2.027462883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:03.024123172 +0000 UTC m=+2388.972779013" watchObservedRunningTime="2026-11-02 20:08:03.027462883 +0000 UTC m=+2388.976118723" Nov 02 20:08:03.129038 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:03.128971 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" podStartSLOduration=2.12895375 podStartE2EDuration="2.12895375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:03.128272703 +0000 UTC m=+2389.076928545" watchObservedRunningTime="2026-11-02 20:08:03.12895375 +0000 UTC m=+2389.077609590" Nov 02 20:08:13.931422 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:13.931391 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:08:13.931909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:13.931463 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:08:14.755096 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:08:14.755055 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:08:14.989601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:14.989573 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:08:15.002957 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:15.002932 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:08:15.005072 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:15.005049 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:08:15.024710 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:15.024643 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:08:18.635440 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:18.635399 4966 scope.go:122] "RemoveContainer" containerID="ddecff5a8f50cac2d1347155732b50e108bb00d0176c6ead83cfc63d4a5cb7e6" Nov 02 20:08:18.647918 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:18.647891 4966 scope.go:122] "RemoveContainer" containerID="a3bf37710cdce76ed30d790e10da166b2b1b17bb5a26370b14cdd7f3ec1a4b06" Nov 02 20:08:18.659970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:18.659950 4966 scope.go:122] "RemoveContainer" containerID="32a049a8d0341cbef522a59939c6bb303fdbbd3c049948ff0d6230b0a6c2a85a" Nov 02 20:08:18.670963 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:18.670940 4966 scope.go:122] "RemoveContainer" containerID="0e0684ab7e042dbc7ce1d43cc6d29bb32729b4d11eb870785a64f4430af011b0" Nov 02 20:08:36.390623 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:36.390579 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb"] Nov 02 20:08:36.433368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:36.433329 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8795bf-5n6vb"] Nov 02 20:08:38.184436 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:38.184384 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1903cc-89cd-4ab9-aed3-a7c22252d8b1" path="/var/lib/kubelet/pods/be1903cc-89cd-4ab9-aed3-a7c22252d8b1/volumes" Nov 02 20:08:43.071402 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.071359 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf"] Nov 02 20:08:43.071981 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.071945 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" containerID="cri-o://8c766fcbb1f7d58389b728c9c11e70e8b9c24741bf0d30b4628f45e186038d0f" gracePeriod=30 Nov 02 20:08:43.086229 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.086194 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z"] Nov 02 20:08:43.086465 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.086439 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" containerID="cri-o://74866f1dab6bf030963a343ebe8dc302de96e215af95ea11c8e52f88edd39aa2" gracePeriod=30 Nov 02 20:08:43.697907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.697869 4966 patch_prober.go:29] interesting pod/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:08:43.698097 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.697941 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:08:43.979142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.979109 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb"] Nov 02 20:08:43.983169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:43.983147 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.018979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.018942 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-dshm\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.019148 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.019032 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-model-cache\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.019148 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.019121 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tls-certs\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.019253 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.019183 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tmp-dir\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.019343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.019289 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fdk2\" (UniqueName: \"kubernetes.io/projected/b1c70c08-8340-44eb-a9cb-711cc360cb3d-kube-api-access-7fdk2\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.019391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.019373 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-home\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.037659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.033020 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-838bc1\"/\"default-dockercfg-2p2j4\"" Nov 02 20:08:44.037659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.036569 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-838bc1\"/\"test-model-838bc1-kserve-self-signed-certs\"" Nov 02 20:08:44.043421 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.043395 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb"] Nov 02 20:08:44.083343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.079363 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-838bc1\"/\"openshift-service-ca.crt\"" Nov 02 20:08:44.083343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.079454 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-838bc1\"/\"kube-root-ca.crt\"" Nov 02 20:08:44.120399 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120366 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-model-cache\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120406 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tls-certs\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120450 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tmp-dir\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120486 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fdk2\" (UniqueName: \"kubernetes.io/projected/b1c70c08-8340-44eb-a9cb-711cc360cb3d-kube-api-access-7fdk2\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120510 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-home\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120850 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120646 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-dshm\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120867 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-model-cache\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120903 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tmp-dir\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.120955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.120941 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-home\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.122821 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.122793 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-dshm\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.123037 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.123021 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tls-certs\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.200348 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.200286 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fdk2\" (UniqueName: \"kubernetes.io/projected/b1c70c08-8340-44eb-a9cb-711cc360cb3d-kube-api-access-7fdk2\") pod \"test-model-838bc1-kserve-7f4556c7cc-npntb\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.296489 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.296399 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:44.708526 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:44.707829 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb"] Nov 02 20:08:44.771829 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:08:44.770900 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:08:45.269600 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:45.269551 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" event={"ID":"b1c70c08-8340-44eb-a9cb-711cc360cb3d","Type":"ContainerStarted","Data":"63712d675863aeaf2ce2b0eaf411b190067c01a97f2b6a10c26e392e792f577e"} Nov 02 20:08:45.269600 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:45.269603 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" event={"ID":"b1c70c08-8340-44eb-a9cb-711cc360cb3d","Type":"ContainerStarted","Data":"a6c156e53a5c4f8570ea7a3ee5a345cf78d49db203d701978e8374d475ee322f"} Nov 02 20:08:45.297491 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:45.297464 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:45.407730 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:45.398487 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" podStartSLOduration=2.39846612 podStartE2EDuration="2.39846612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:45.392613046 +0000 UTC m=+2431.341268891" watchObservedRunningTime="2026-11-02 20:08:45.39846612 +0000 UTC m=+2431.347121960" Nov 02 20:08:48.302810 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.302775 4966 generic.go:356] "Generic (PLEG): container finished" podID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerID="74866f1dab6bf030963a343ebe8dc302de96e215af95ea11c8e52f88edd39aa2" exitCode=0 Nov 02 20:08:48.303326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.303019 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" event={"ID":"90683c29-991f-4ed9-87e9-05d7b79d7aa1","Type":"ContainerDied","Data":"74866f1dab6bf030963a343ebe8dc302de96e215af95ea11c8e52f88edd39aa2"} Nov 02 20:08:48.305066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.305036 4966 generic.go:356] "Generic (PLEG): container finished" podID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerID="8c766fcbb1f7d58389b728c9c11e70e8b9c24741bf0d30b4628f45e186038d0f" exitCode=0 Nov 02 20:08:48.305208 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.305076 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" event={"ID":"80b8cb02-566c-4175-a961-c2d8a1f25316","Type":"ContainerDied","Data":"8c766fcbb1f7d58389b728c9c11e70e8b9c24741bf0d30b4628f45e186038d0f"} Nov 02 20:08:48.388371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.388337 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 20:08:48.460515 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.460481 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 20:08:48.546028 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.545992 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-envoy\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546232 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546037 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-data\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546232 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546106 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-token\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-token\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546232 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546144 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-socket\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546232 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546167 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b8cb02-566c-4175-a961-c2d8a1f25316-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b8cb02-566c-4175-a961-c2d8a1f25316-istiod-ca-cert\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546476 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546314 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-credential-socket\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546476 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546393 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-podinfo\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546476 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546426 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-kube-api-access-w8b74\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-kube-api-access-w8b74\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.546624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.546477 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-certs\") pod \"80b8cb02-566c-4175-a961-c2d8a1f25316\" (UID: \"80b8cb02-566c-4175-a961-c2d8a1f25316\") " Nov 02 20:08:48.547257 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.547231 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-certs" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.548018 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.547713 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b8cb02-566c-4175-a961-c2d8a1f25316-istiod-ca-cert" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:48.548018 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.548010 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-socket" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.548246 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.548214 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-data" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.548465 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.548418 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-credential-socket" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.551485 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.551454 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-podinfo" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:08:48.551622 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.551592 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-kube-api-access-w8b74" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "kube-api-access-w8b74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.551743 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.551625 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-envoy" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.552474 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.552454 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-token" pod "80b8cb02-566c-4175-a961-c2d8a1f25316" (UID: "80b8cb02-566c-4175-a961-c2d8a1f25316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.647846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.647761 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istiod-ca-cert\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.647846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.647803 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-kube-api-access-vhljp\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-kube-api-access-vhljp\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.647846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.647834 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-credential-socket\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.648162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.647954 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-certs\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.648162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648039 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-podinfo\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.648162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648124 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-envoy\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.648362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648179 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-data\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.648362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648230 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-token\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-token\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.648362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648270 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-socket\") pod \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\" (UID: \"90683c29-991f-4ed9-87e9-05d7b79d7aa1\") " Nov 02 20:08:48.648362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648283 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-certs" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.648362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648293 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-credential-socket" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.648362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648338 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istiod-ca-cert" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:48.648701 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648542 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-data" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.648701 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.648571 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-socket" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.649102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649076 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649163 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649108 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649163 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649123 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649163 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649136 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649163 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649150 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649164 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649178 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649191 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b8cb02-566c-4175-a961-c2d8a1f25316-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649204 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649217 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649231 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649244 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b8cb02-566c-4175-a961-c2d8a1f25316-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649259 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b8cb02-566c-4175-a961-c2d8a1f25316-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.649293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.649271 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8b74\" (UniqueName: \"kubernetes.io/projected/80b8cb02-566c-4175-a961-c2d8a1f25316-kube-api-access-w8b74\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.650855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.650721 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-envoy" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:48.650855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.650791 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-token" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.651168 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.651144 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-kube-api-access-vhljp" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "kube-api-access-vhljp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:48.651584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.651563 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-podinfo" pod "90683c29-991f-4ed9-87e9-05d7b79d7aa1" (UID: "90683c29-991f-4ed9-87e9-05d7b79d7aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:08:48.750516 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.750478 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.750516 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.750514 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.750761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.750527 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:48.750761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:48.750540 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhljp\" (UniqueName: \"kubernetes.io/projected/90683c29-991f-4ed9-87e9-05d7b79d7aa1-kube-api-access-vhljp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:49.315064 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.315029 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" Nov 02 20:08:49.315568 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.315065 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z" event={"ID":"90683c29-991f-4ed9-87e9-05d7b79d7aa1","Type":"ContainerDied","Data":"5cb7dd06e248b19a462cac10e09f3dd4dc798c0aed060c4ebd8ceeb5fc48b3c8"} Nov 02 20:08:49.315568 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.315137 4966 scope.go:122] "RemoveContainer" containerID="74866f1dab6bf030963a343ebe8dc302de96e215af95ea11c8e52f88edd39aa2" Nov 02 20:08:49.317774 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.317752 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" Nov 02 20:08:49.317875 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.317766 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf" event={"ID":"80b8cb02-566c-4175-a961-c2d8a1f25316","Type":"ContainerDied","Data":"e58e6edbf0ca994599cbf8d4211adfdba021a88e9e3290ca346f0aafa0af22c9"} Nov 02 20:08:49.349473 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.349430 4966 scope.go:122] "RemoveContainer" containerID="8c766fcbb1f7d58389b728c9c11e70e8b9c24741bf0d30b4628f45e186038d0f" Nov 02 20:08:49.475772 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.475738 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z"] Nov 02 20:08:49.521633 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.521595 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-l9l6z"] Nov 02 20:08:49.580251 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.580201 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf"] Nov 02 20:08:49.583868 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:49.583837 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8795bf-openshift-default-c67bc566c-2wzrf"] Nov 02 20:08:50.195349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:50.190716 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" path="/var/lib/kubelet/pods/80b8cb02-566c-4175-a961-c2d8a1f25316/volumes" Nov 02 20:08:50.195349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:50.192030 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" path="/var/lib/kubelet/pods/90683c29-991f-4ed9-87e9-05d7b79d7aa1/volumes" Nov 02 20:08:54.297288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:54.297251 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:54.311021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:54.310985 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:08:54.380335 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:08:54.380290 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:09:12.116067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:12.116022 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758"] Nov 02 20:09:12.116603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:12.116282 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" containerID="cri-o://3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae" gracePeriod=30 Nov 02 20:09:12.226552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:12.226491 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx"] Nov 02 20:09:12.227082 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:12.227049 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" containerID="cri-o://44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6" gracePeriod=30 Nov 02 20:09:13.143681 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:13.143634 4966 patch_prober.go:29] interesting pod/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:09:13.144097 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:13.143713 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:09:14.380865 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:14.380738 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:09:14.380865 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:14.380822 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:09:14.752960 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:09:14.752918 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:09:17.423678 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.423644 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 20:09:17.499124 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.499096 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 20:09:17.535093 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535058 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-token\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-token\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535093 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535099 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-certs\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535392 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535161 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-credential-socket\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535392 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535254 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-podinfo\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535392 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535354 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-data\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.535561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535400 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-data\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535440 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-podinfo\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.535561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535505 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-certs" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.535561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535517 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-credential-socket" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.535561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535512 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-kube-api-access-pvhbv\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-kube-api-access-pvhbv\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535599 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-token\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-token\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535633 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-socket\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535647 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-data" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535691 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-socket\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535728 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-envoy\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535788 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-kube-api-access-sqdvh\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-kube-api-access-sqdvh\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535827 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a22f389-14eb-45aa-b240-debdc9244ebf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22f389-14eb-45aa-b240-debdc9244ebf-istiod-ca-cert\") pod \"1a22f389-14eb-45aa-b240-debdc9244ebf\" (UID: \"1a22f389-14eb-45aa-b240-debdc9244ebf\") " Nov 02 20:09:17.535897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535859 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istiod-ca-cert\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.536408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535915 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-envoy\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.536408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535971 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-credential-socket\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.536408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.536004 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-certs\") pod \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\" (UID: \"0b367fbd-ccfe-4cc6-8771-4ee083ae0422\") " Nov 02 20:09:17.536643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.536620 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.536721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535729 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-data" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.536721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.535907 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-socket" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.536721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.536335 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-socket" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.536721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.536653 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.536721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.536711 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.536995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.536929 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a22f389-14eb-45aa-b240-debdc9244ebf-istiod-ca-cert" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:17.536995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.536979 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istiod-ca-cert" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:17.537246 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.537221 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-credential-socket" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.537383 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.537361 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-certs" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.538233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.538206 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-podinfo" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:09:17.538467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.538441 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-kube-api-access-sqdvh" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "kube-api-access-sqdvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:17.538902 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.538866 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-kube-api-access-pvhbv" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "kube-api-access-pvhbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:17.539110 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.539086 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-podinfo" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:09:17.539278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.539202 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-token" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:17.539398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.539379 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-envoy" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.539541 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.539520 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-token" pod "0b367fbd-ccfe-4cc6-8771-4ee083ae0422" (UID: "0b367fbd-ccfe-4cc6-8771-4ee083ae0422"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:17.540096 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.540077 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-envoy" pod "1a22f389-14eb-45aa-b240-debdc9244ebf" (UID: "1a22f389-14eb-45aa-b240-debdc9244ebf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:17.568165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.568134 4966 generic.go:356] "Generic (PLEG): container finished" podID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerID="3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae" exitCode=0 Nov 02 20:09:17.568401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.568209 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" Nov 02 20:09:17.568401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.568219 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" event={"ID":"0b367fbd-ccfe-4cc6-8771-4ee083ae0422","Type":"ContainerDied","Data":"3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae"} Nov 02 20:09:17.568401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.568343 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758" event={"ID":"0b367fbd-ccfe-4cc6-8771-4ee083ae0422","Type":"ContainerDied","Data":"6719392bc388f4df008332169e093fdb83f87e9719e66e1468d91281d7fecec3"} Nov 02 20:09:17.568401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.568384 4966 scope.go:122] "RemoveContainer" containerID="3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae" Nov 02 20:09:17.570072 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.570045 4966 generic.go:356] "Generic (PLEG): container finished" podID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerID="44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6" exitCode=0 Nov 02 20:09:17.570253 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.570237 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" event={"ID":"1a22f389-14eb-45aa-b240-debdc9244ebf","Type":"ContainerDied","Data":"44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6"} Nov 02 20:09:17.570367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.570280 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" event={"ID":"1a22f389-14eb-45aa-b240-debdc9244ebf","Type":"ContainerDied","Data":"0fe372103b2b83deb861258279d797e1b566253ae9a85f5203df745129c13ee1"} Nov 02 20:09:17.570491 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.570403 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx" Nov 02 20:09:17.592467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.592436 4966 scope.go:122] "RemoveContainer" containerID="3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae" Nov 02 20:09:17.593138 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:09:17.593114 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae\": container with ID starting with 3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae not found: ID does not exist" containerID="3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae" Nov 02 20:09:17.593253 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.593146 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae"} err="failed to get container status \"3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae\": rpc error: code = NotFound desc = could not find container \"3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae\": container with ID starting with 3df97971b8aadf400a66081570569f24614cf0f2315fa90a4712e0358ab5fcae not found: ID does not exist" Nov 02 20:09:17.593253 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.593165 4966 scope.go:122] "RemoveContainer" containerID="44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6" Nov 02 20:09:17.607037 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.607010 4966 scope.go:122] "RemoveContainer" containerID="44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6" Nov 02 20:09:17.607425 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:09:17.607395 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6\": container with ID starting with 44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6 not found: ID does not exist" containerID="44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6" Nov 02 20:09:17.607506 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.607439 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6"} err="failed to get container status \"44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6\": rpc error: code = NotFound desc = could not find container \"44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6\": container with ID starting with 44bd287af4eeaa5ccfed2224c131f31397d0841efb0249ec6c842b262b7626b6 not found: ID does not exist" Nov 02 20:09:17.638053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638016 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638050 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvhbv\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-kube-api-access-pvhbv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638065 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638077 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638090 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638101 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638113 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqdvh\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-kube-api-access-sqdvh\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638125 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a22f389-14eb-45aa-b240-debdc9244ebf-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638137 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638162 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638175 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638187 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b367fbd-ccfe-4cc6-8771-4ee083ae0422-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638200 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638212 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.638289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.638224 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a22f389-14eb-45aa-b240-debdc9244ebf-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:17.841830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.841790 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx"] Nov 02 20:09:17.847542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.847509 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b2ssx"] Nov 02 20:09:17.946799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.946755 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758"] Nov 02 20:09:17.950512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:17.950481 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-9s758"] Nov 02 20:09:18.187709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:18.187668 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" path="/var/lib/kubelet/pods/0b367fbd-ccfe-4cc6-8771-4ee083ae0422/volumes" Nov 02 20:09:18.188443 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:18.188419 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" path="/var/lib/kubelet/pods/1a22f389-14eb-45aa-b240-debdc9244ebf/volumes" Nov 02 20:09:18.720032 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:18.719994 4966 scope.go:122] "RemoveContainer" containerID="20e4fd80a95dfc190668e70b2bcfe297640221eb6773ac6a74c1fe74da720955" Nov 02 20:09:18.730887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:18.730867 4966 scope.go:122] "RemoveContainer" containerID="b6e9eae49bdc6c9dc5000e4fbcd572e388252ab2b445dc56873718ed4d4ad864" Nov 02 20:09:18.740967 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:18.740944 4966 scope.go:122] "RemoveContainer" containerID="a47dd1d3225335b5272fd1968026cb851f857a3de85ee42572669f3344e425e8" Nov 02 20:09:33.365218 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365130 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw"] Nov 02 20:09:33.365742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365684 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" Nov 02 20:09:33.365742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365696 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" Nov 02 20:09:33.365742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365725 4966 cpu_manager.go:414] "containerMap: removing container" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" Nov 02 20:09:33.365742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365730 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" Nov 02 20:09:33.365742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365742 4966 cpu_manager.go:414] "containerMap: removing container" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" Nov 02 20:09:33.365922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365747 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" Nov 02 20:09:33.365922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365764 4966 cpu_manager.go:414] "containerMap: removing container" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" Nov 02 20:09:33.365922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365770 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" Nov 02 20:09:33.365922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365864 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a22f389-14eb-45aa-b240-debdc9244ebf" containerName="istio-proxy" Nov 02 20:09:33.365922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365876 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b8cb02-566c-4175-a961-c2d8a1f25316" containerName="istio-proxy" Nov 02 20:09:33.365922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365885 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b367fbd-ccfe-4cc6-8771-4ee083ae0422" containerName="istio-proxy" Nov 02 20:09:33.365922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.365893 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="90683c29-991f-4ed9-87e9-05d7b79d7aa1" containerName="istio-proxy" Nov 02 20:09:33.371115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.371094 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.425117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.425054 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw"] Nov 02 20:09:33.465452 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.465418 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-28fa24\"/\"openshift-service-ca.crt\"" Nov 02 20:09:33.465633 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.465458 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-28fa24\"/\"kube-root-ca.crt\"" Nov 02 20:09:33.467151 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.467135 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-28fa24\"/\"default-dockercfg-pfk5w\"" Nov 02 20:09:33.467202 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.467143 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-28fa24\"/\"test-model-28fa24-kserve-self-signed-certs\"" Nov 02 20:09:33.544486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.544438 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ae073c-a362-4c94-84db-8b6ad37e1183-tls-certs\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.544718 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.544598 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-home\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.544718 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.544650 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-model-cache\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.544718 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.544679 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-tmp-dir\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.544939 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.544736 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxlr\" (UniqueName: \"kubernetes.io/projected/43ae073c-a362-4c94-84db-8b6ad37e1183-kube-api-access-kgxlr\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.544939 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.544767 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-dshm\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.645938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.645838 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ae073c-a362-4c94-84db-8b6ad37e1183-tls-certs\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.645980 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-home\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.646031 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-model-cache\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646256 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.646167 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-tmp-dir\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.646262 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxlr\" (UniqueName: \"kubernetes.io/projected/43ae073c-a362-4c94-84db-8b6ad37e1183-kube-api-access-kgxlr\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.646312 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-dshm\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646460 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.646433 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-model-cache\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.646489 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-home\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.646519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.646511 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-tmp-dir\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.648587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.648553 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-dshm\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.648587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.648567 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ae073c-a362-4c94-84db-8b6ad37e1183-tls-certs\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.700154 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.700098 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxlr\" (UniqueName: \"kubernetes.io/projected/43ae073c-a362-4c94-84db-8b6ad37e1183-kube-api-access-kgxlr\") pod \"test-model-28fa24-kserve-7d4d54484f-vnkfw\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:33.984525 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:33.984489 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:34.366579 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:34.366546 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw"] Nov 02 20:09:34.367949 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:09:34.367922 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43ae073c_a362_4c94_84db_8b6ad37e1183.slice/crio-6674e27c70a2bcea0abae8b66e859a1537ba057c20655c6c0df4657ff5ffe525 WatchSource:0}: Error finding container 6674e27c70a2bcea0abae8b66e859a1537ba057c20655c6c0df4657ff5ffe525: Status 404 returned error can't find the container with id 6674e27c70a2bcea0abae8b66e859a1537ba057c20655c6c0df4657ff5ffe525 Nov 02 20:09:34.703838 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:34.703737 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" event={"ID":"43ae073c-a362-4c94-84db-8b6ad37e1183","Type":"ContainerStarted","Data":"65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e"} Nov 02 20:09:34.703838 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:34.703783 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" event={"ID":"43ae073c-a362-4c94-84db-8b6ad37e1183","Type":"ContainerStarted","Data":"6674e27c70a2bcea0abae8b66e859a1537ba057c20655c6c0df4657ff5ffe525"} Nov 02 20:09:34.816068 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:34.815996 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" podStartSLOduration=1.815977051 podStartE2EDuration="1.815977051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:34.810091794 +0000 UTC m=+2480.758747631" watchObservedRunningTime="2026-11-02 20:09:34.815977051 +0000 UTC m=+2480.764632890" Nov 02 20:09:34.985336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:34.985279 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:43.985504 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:43.985461 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:44.004713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.004682 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:44.669355 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.669311 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6"] Nov 02 20:09:44.673546 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.673519 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.730594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.730560 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6"] Nov 02 20:09:44.760138 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:09:44.760100 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:09:44.771067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.771036 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19d134\"/\"test-model-19d134-kserve-self-signed-certs\"" Nov 02 20:09:44.771067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.771037 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19d134\"/\"openshift-service-ca.crt\"" Nov 02 20:09:44.771285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.771051 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19d134\"/\"kube-root-ca.crt\"" Nov 02 20:09:44.771285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.771086 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19d134\"/\"default-dockercfg-skh2s\"" Nov 02 20:09:44.778244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.778220 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-dshm\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.778406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.778265 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5bf48b0-4073-420d-83b0-4f577342622e-tls-certs\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.778491 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.778464 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-model-cache\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.778554 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.778542 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-home\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.778664 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.778638 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-tmp-dir\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.778720 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.778688 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8846p\" (UniqueName: \"kubernetes.io/projected/e5bf48b0-4073-420d-83b0-4f577342622e-kube-api-access-8846p\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.825935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.825900 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:09:44.879690 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.879644 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8846p\" (UniqueName: \"kubernetes.io/projected/e5bf48b0-4073-420d-83b0-4f577342622e-kube-api-access-8846p\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.879904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.879718 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-dshm\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.879904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.879765 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5bf48b0-4073-420d-83b0-4f577342622e-tls-certs\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.879904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.879870 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-model-cache\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.880096 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.879908 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-home\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.880096 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.879941 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-tmp-dir\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.880571 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.880291 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-tmp-dir\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.880571 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.880536 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-home\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.880784 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.880535 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-model-cache\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.883571 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.883547 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5bf48b0-4073-420d-83b0-4f577342622e-tls-certs\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.886581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.886552 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-dshm\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.936500 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.936429 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8846p\" (UniqueName: \"kubernetes.io/projected/e5bf48b0-4073-420d-83b0-4f577342622e-kube-api-access-8846p\") pod \"test-model-19d134-kserve-8595896f7f-xcqt6\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:44.990722 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:44.990683 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:45.383103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:45.383057 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6"] Nov 02 20:09:45.385283 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:09:45.385255 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5bf48b0_4073_420d_83b0_4f577342622e.slice/crio-0b69ea997e1c98277b1d5d0dd90933069e152ead241b1f03ddcf66006cf3965c WatchSource:0}: Error finding container 0b69ea997e1c98277b1d5d0dd90933069e152ead241b1f03ddcf66006cf3965c: Status 404 returned error can't find the container with id 0b69ea997e1c98277b1d5d0dd90933069e152ead241b1f03ddcf66006cf3965c Nov 02 20:09:45.817112 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:45.817071 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" event={"ID":"e5bf48b0-4073-420d-83b0-4f577342622e","Type":"ContainerStarted","Data":"8ea133c3c27f6ec129f88bdaeaf8f8d068f607c89cce95771bc217bc0b8a34c0"} Nov 02 20:09:45.817112 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:45.817125 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" event={"ID":"e5bf48b0-4073-420d-83b0-4f577342622e","Type":"ContainerStarted","Data":"0b69ea997e1c98277b1d5d0dd90933069e152ead241b1f03ddcf66006cf3965c"} Nov 02 20:09:45.926727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:45.926656 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" podStartSLOduration=1.926635281 podStartE2EDuration="1.926635281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:45.922887756 +0000 UTC m=+2491.871543596" watchObservedRunningTime="2026-11-02 20:09:45.926635281 +0000 UTC m=+2491.875291122" Nov 02 20:09:45.992067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:45.992026 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:54.991220 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:54.991179 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:55.003962 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:55.003928 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:09:55.911190 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:09:55.911157 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:10:14.754511 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:10:14.754470 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:10:18.796824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:18.796789 4966 scope.go:122] "RemoveContainer" containerID="62324e96e336196597a6ccd33ba7f1526db4161bc0b49aca3932e647b7ef0029" Nov 02 20:10:18.807290 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:18.807267 4966 scope.go:122] "RemoveContainer" containerID="d79bcc50e4880021cb3911de794e74eadc91529d9685ca538180e4b4d11b5e54" Nov 02 20:10:18.818872 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:18.818841 4966 scope.go:122] "RemoveContainer" containerID="9d315bbad194a47868b3ab719a8ca50723a9b3a7ca2b246cdac99c05eb65f700" Nov 02 20:10:37.466169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.466129 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr"] Nov 02 20:10:37.472822 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.472797 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.523916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.523868 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr"] Nov 02 20:10:37.550388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.550349 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxmn\" (UniqueName: \"kubernetes.io/projected/c24a2a7e-0959-4760-9b2e-86681db307dd-kube-api-access-wtxmn\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.550614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.550493 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.550614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.550530 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.565466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.565430 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:10:37.565675 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.565477 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:10:37.567095 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.567057 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:10:37.567234 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.567135 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:10:37.651533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.651488 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxmn\" (UniqueName: \"kubernetes.io/projected/c24a2a7e-0959-4760-9b2e-86681db307dd-kube-api-access-wtxmn\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.651742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.651599 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.651742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.651639 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.652352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.652325 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.654162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.654139 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.704909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.704873 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxmn\" (UniqueName: \"kubernetes.io/projected/c24a2a7e-0959-4760-9b2e-86681db307dd-kube-api-access-wtxmn\") pod \"maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:37.789680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:37.789650 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:38.169929 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:38.169837 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr"] Nov 02 20:10:38.227860 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:38.227828 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" event={"ID":"c24a2a7e-0959-4760-9b2e-86681db307dd","Type":"ContainerStarted","Data":"a28087beb7dab67087d92b19a55ac41c4e129f6062e9d22b51e18bc5924ea202"} Nov 02 20:10:39.237581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:39.237546 4966 generic.go:356] "Generic (PLEG): container finished" podID="c24a2a7e-0959-4760-9b2e-86681db307dd" containerID="72f71525dc64ae24f778164e75417c50134a771817fd2c1ad0b9202af6f17ef9" exitCode=0 Nov 02 20:10:39.237986 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:39.237623 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" event={"ID":"c24a2a7e-0959-4760-9b2e-86681db307dd","Type":"ContainerDied","Data":"72f71525dc64ae24f778164e75417c50134a771817fd2c1ad0b9202af6f17ef9"} Nov 02 20:10:40.415844 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.415818 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:40.483590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.483561 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-xks-ca\") pod \"c24a2a7e-0959-4760-9b2e-86681db307dd\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " Nov 02 20:10:40.483741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.483622 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24a2a7e-0959-4760-9b2e-86681db307dd-kube-api-access-wtxmn\" (UniqueName: \"kubernetes.io/projected/c24a2a7e-0959-4760-9b2e-86681db307dd-kube-api-access-wtxmn\") pod \"c24a2a7e-0959-4760-9b2e-86681db307dd\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " Nov 02 20:10:40.483741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.483732 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-openshift-ca\") pod \"c24a2a7e-0959-4760-9b2e-86681db307dd\" (UID: \"c24a2a7e-0959-4760-9b2e-86681db307dd\") " Nov 02 20:10:40.484104 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.484075 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-openshift-ca" pod "c24a2a7e-0959-4760-9b2e-86681db307dd" (UID: "c24a2a7e-0959-4760-9b2e-86681db307dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:40.485683 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.485661 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-xks-ca" pod "c24a2a7e-0959-4760-9b2e-86681db307dd" (UID: "c24a2a7e-0959-4760-9b2e-86681db307dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:10:40.485803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.485772 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24a2a7e-0959-4760-9b2e-86681db307dd-kube-api-access-wtxmn" pod "c24a2a7e-0959-4760-9b2e-86681db307dd" (UID: "c24a2a7e-0959-4760-9b2e-86681db307dd"). InnerVolumeSpecName "kube-api-access-wtxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:40.585197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.585096 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxmn\" (UniqueName: \"kubernetes.io/projected/c24a2a7e-0959-4760-9b2e-86681db307dd-kube-api-access-wtxmn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:40.585197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.585138 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:40.585197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:40.585167 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24a2a7e-0959-4760-9b2e-86681db307dd-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:41.254676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:41.254641 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" Nov 02 20:10:41.254873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:41.254663 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr" event={"ID":"c24a2a7e-0959-4760-9b2e-86681db307dd","Type":"ContainerDied","Data":"a28087beb7dab67087d92b19a55ac41c4e129f6062e9d22b51e18bc5924ea202"} Nov 02 20:10:41.254873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:41.254729 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a28087beb7dab67087d92b19a55ac41c4e129f6062e9d22b51e18bc5924ea202" Nov 02 20:10:44.753614 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:10:44.753574 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:10:54.048639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:54.048600 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg"] Nov 02 20:10:54.067447 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:54.067413 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg"] Nov 02 20:10:54.094220 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:54.094181 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59461aa9-rr4fg"] Nov 02 20:10:54.097364 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:54.097342 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2362d11a-6dcfg"] Nov 02 20:10:54.183763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:54.183710 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392c2aac-340b-4a15-93b2-f3ff7179f44d" path="/var/lib/kubelet/pods/392c2aac-340b-4a15-93b2-f3ff7179f44d/volumes" Nov 02 20:10:54.184388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:10:54.184366 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b8a94f-a0a3-4387-82af-d70c20d0d604" path="/var/lib/kubelet/pods/d7b8a94f-a0a3-4387-82af-d70c20d0d604/volumes" Nov 02 20:11:02.044319 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:02.044214 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g"] Nov 02 20:11:02.089937 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:02.089902 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1ff980-ldx8g"] Nov 02 20:11:02.186996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:02.186951 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989c00b9-29a1-448d-81cf-7e76c5b254ee" path="/var/lib/kubelet/pods/989c00b9-29a1-448d-81cf-7e76c5b254ee/volumes" Nov 02 20:11:03.073777 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:03.073734 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj"] Nov 02 20:11:03.117916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:03.117878 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22e7c9c-lmqlj"] Nov 02 20:11:04.183644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:04.183608 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285f6908-ff18-4259-b37f-b0a4ea9acb64" path="/var/lib/kubelet/pods/285f6908-ff18-4259-b37f-b0a4ea9acb64/volumes" Nov 02 20:11:14.752915 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:11:14.752872 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:11:18.871310 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:18.871259 4966 scope.go:122] "RemoveContainer" containerID="85bfa4ac0bb1f9633715b5bae983df57fc01b7ed374f4bf4f11999ba44180f3a" Nov 02 20:11:44.755126 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:11:44.755086 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:11:49.859352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:49.859293 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472"] Nov 02 20:11:49.859809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:49.859593 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" podUID="30ea0192-0399-4102-b8de-da2709237b95" containerName="discovery" containerID="cri-o://5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89" gracePeriod=30 Nov 02 20:11:50.144196 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.144163 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 20:11:50.234408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234374 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-dns-cert\") pod \"30ea0192-0399-4102-b8de-da2709237b95\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " Nov 02 20:11:50.234607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234420 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-kubeconfig\") pod \"30ea0192-0399-4102-b8de-da2709237b95\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " Nov 02 20:11:50.234607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234487 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-ca-configmap\") pod \"30ea0192-0399-4102-b8de-da2709237b95\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " Nov 02 20:11:50.234607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234521 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-kube-api-access-2vpw6\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-kube-api-access-2vpw6\") pod \"30ea0192-0399-4102-b8de-da2709237b95\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " Nov 02 20:11:50.234607 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234591 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-cacerts\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-cacerts\") pod \"30ea0192-0399-4102-b8de-da2709237b95\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " Nov 02 20:11:50.234835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234650 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-istio-token\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-istio-token\") pod \"30ea0192-0399-4102-b8de-da2709237b95\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " Nov 02 20:11:50.234835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234788 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ea0192-0399-4102-b8de-da2709237b95-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea0192-0399-4102-b8de-da2709237b95-local-certs\") pod \"30ea0192-0399-4102-b8de-da2709237b95\" (UID: \"30ea0192-0399-4102-b8de-da2709237b95\") " Nov 02 20:11:50.234968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.234939 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-ca-configmap" pod "30ea0192-0399-4102-b8de-da2709237b95" (UID: "30ea0192-0399-4102-b8de-da2709237b95"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:11:50.235412 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.235387 4966 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-ca-configmap\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:50.236996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.236955 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-cacerts" pod "30ea0192-0399-4102-b8de-da2709237b95" (UID: "30ea0192-0399-4102-b8de-da2709237b95"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:11:50.237280 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.237253 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-kubeconfig" pod "30ea0192-0399-4102-b8de-da2709237b95" (UID: "30ea0192-0399-4102-b8de-da2709237b95"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:11:50.238330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.237587 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-kube-api-access-2vpw6" pod "30ea0192-0399-4102-b8de-da2709237b95" (UID: "30ea0192-0399-4102-b8de-da2709237b95"). InnerVolumeSpecName "kube-api-access-2vpw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:11:50.238330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.237631 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-istio-token" pod "30ea0192-0399-4102-b8de-da2709237b95" (UID: "30ea0192-0399-4102-b8de-da2709237b95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:11:50.238330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.237642 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ea0192-0399-4102-b8de-da2709237b95-local-certs" pod "30ea0192-0399-4102-b8de-da2709237b95" (UID: "30ea0192-0399-4102-b8de-da2709237b95"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:50.238330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.237817 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-dns-cert" pod "30ea0192-0399-4102-b8de-da2709237b95" (UID: "30ea0192-0399-4102-b8de-da2709237b95"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:11:50.337185 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.337146 4966 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-cacerts\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:50.337185 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.337178 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:50.337497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.337193 4966 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea0192-0399-4102-b8de-da2709237b95-local-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:50.337497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.337221 4966 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-csr-dns-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:50.337497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.337231 4966 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30ea0192-0399-4102-b8de-da2709237b95-istio-kubeconfig\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:50.337497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.337242 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vpw6\" (UniqueName: \"kubernetes.io/projected/30ea0192-0399-4102-b8de-da2709237b95-kube-api-access-2vpw6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:50.786992 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.786953 4966 generic.go:356] "Generic (PLEG): container finished" podID="30ea0192-0399-4102-b8de-da2709237b95" containerID="5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89" exitCode=0 Nov 02 20:11:50.787204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.787017 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" Nov 02 20:11:50.787204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.787035 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" event={"ID":"30ea0192-0399-4102-b8de-da2709237b95","Type":"ContainerDied","Data":"5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89"} Nov 02 20:11:50.787204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.787092 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472" event={"ID":"30ea0192-0399-4102-b8de-da2709237b95","Type":"ContainerDied","Data":"8c0537539c4383d200431cee941854f66391ea000c1c26c1b44ecd4dc9383424"} Nov 02 20:11:50.787204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.787112 4966 scope.go:122] "RemoveContainer" containerID="5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89" Nov 02 20:11:50.804740 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.804375 4966 scope.go:122] "RemoveContainer" containerID="5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89" Nov 02 20:11:50.804938 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:11:50.804914 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89\": container with ID starting with 5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89 not found: ID does not exist" containerID="5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89" Nov 02 20:11:50.804983 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.804954 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89"} err="failed to get container status \"5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89\": rpc error: code = NotFound desc = could not find container \"5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89\": container with ID starting with 5c503831984092f1a828bfc3a9793f57c072daa936a70c69aee524a31f2aeb89 not found: ID does not exist" Nov 02 20:11:50.988608 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:50.988570 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472"] Nov 02 20:11:51.032790 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:51.032742 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7q472"] Nov 02 20:11:52.184188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:11:52.184151 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ea0192-0399-4102-b8de-da2709237b95" path="/var/lib/kubelet/pods/30ea0192-0399-4102-b8de-da2709237b95/volumes" Nov 02 20:12:14.752133 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:12:14.752095 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:12:18.900531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:18.900495 4966 scope.go:122] "RemoveContainer" containerID="591ceb63830c1ea4b0076763ce045b9866951ca95bedeba031774479a68120c0" Nov 02 20:12:18.913467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:18.913446 4966 scope.go:122] "RemoveContainer" containerID="4edd0129c9a8b9f82dd2f317c02ec98f3a7ba9d90d8e0e4a397667484c6249e1" Nov 02 20:12:18.922737 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:18.922714 4966 scope.go:122] "RemoveContainer" containerID="ff26f95e866129d7e2fab8f9c9711b667b61f80a2e77f308352d781fd58e4a00" Nov 02 20:12:18.932000 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:18.931977 4966 scope.go:122] "RemoveContainer" containerID="cf553e7fa33c5a7371eb5d19c440036f8e9969faf8bef93801da7dad34f4b133" Nov 02 20:12:18.940722 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:18.940701 4966 scope.go:122] "RemoveContainer" containerID="5e2013e1a341f2f35970ab718944d3a08bd97bbd11c7d96d7b7bcb99a1b6f091" Nov 02 20:12:18.953070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:18.953046 4966 scope.go:122] "RemoveContainer" containerID="7cf4993d32fd4a0980658c9b4d82ad9b364b6c59c33135c10d74fc59674a1bb5" Nov 02 20:12:18.963278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:18.963259 4966 scope.go:122] "RemoveContainer" containerID="bc643b0076e1207847a3c24fb2e9ddf71766846fda6af5c4c0d3d39f9359fd0c" Nov 02 20:12:22.046870 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:22.046830 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk"] Nov 02 20:12:22.051931 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:22.051902 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24"] Nov 02 20:12:22.056641 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:22.056614 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49806dc8-9vzgk"] Nov 02 20:12:22.061062 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:22.061029 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9a48a3-jjl24"] Nov 02 20:12:22.183595 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:22.183565 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554e692d-c8a4-4b6b-bef1-2e2a4313c9ec" path="/var/lib/kubelet/pods/554e692d-c8a4-4b6b-bef1-2e2a4313c9ec/volumes" Nov 02 20:12:22.184033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:22.184016 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88992671-b828-4f6d-818f-16cc845d843c" path="/var/lib/kubelet/pods/88992671-b828-4f6d-818f-16cc845d843c/volumes" Nov 02 20:12:36.409701 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:36.409623 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr"] Nov 02 20:12:36.455772 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:36.455743 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1db0e4-6p6hr"] Nov 02 20:12:38.186837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:38.186801 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24a2a7e-0959-4760-9b2e-86681db307dd" path="/var/lib/kubelet/pods/c24a2a7e-0959-4760-9b2e-86681db307dd/volumes" Nov 02 20:12:43.558214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.558179 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs"] Nov 02 20:12:43.558814 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.558797 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c24a2a7e-0959-4760-9b2e-86681db307dd" containerName="revoke-keys" Nov 02 20:12:43.558863 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.558816 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24a2a7e-0959-4760-9b2e-86681db307dd" containerName="revoke-keys" Nov 02 20:12:43.558863 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.558862 4966 cpu_manager.go:414] "containerMap: removing container" podUID="30ea0192-0399-4102-b8de-da2709237b95" containerName="discovery" Nov 02 20:12:43.558933 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.558867 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ea0192-0399-4102-b8de-da2709237b95" containerName="discovery" Nov 02 20:12:43.558968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.558952 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ea0192-0399-4102-b8de-da2709237b95" containerName="discovery" Nov 02 20:12:43.558968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.558961 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24a2a7e-0959-4760-9b2e-86681db307dd" containerName="revoke-keys" Nov 02 20:12:43.562460 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.562445 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.610723 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.610691 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:12:43.656922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.656886 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:12:43.657106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.657042 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:12:43.660544 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.660517 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs"] Nov 02 20:12:43.671621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.671589 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.671753 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.671681 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwbp\" (UniqueName: \"kubernetes.io/projected/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-kube-api-access-xzwbp\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.671753 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.671708 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.710713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.710689 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:12:43.772772 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.772742 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.772916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.772833 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwbp\" (UniqueName: \"kubernetes.io/projected/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-kube-api-access-xzwbp\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.772916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.772874 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.773473 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.773447 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.775326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.775287 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.825357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.825276 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwbp\" (UniqueName: \"kubernetes.io/projected/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-kube-api-access-xzwbp\") pod \"maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:43.877565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:43.877530 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:44.245499 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:44.245460 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs"] Nov 02 20:12:44.246659 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:12:44.246621 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9b3a2d_d165_48b5_b31c_5f33ab80d70e.slice/crio-1057c04d2e54b025b0df6cc47c91da5cddcfd24513c346fa0ab9fc8684dd0edd WatchSource:0}: Error finding container 1057c04d2e54b025b0df6cc47c91da5cddcfd24513c346fa0ab9fc8684dd0edd: Status 404 returned error can't find the container with id 1057c04d2e54b025b0df6cc47c91da5cddcfd24513c346fa0ab9fc8684dd0edd Nov 02 20:12:44.758618 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:12:44.758573 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:12:45.202967 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:45.202935 4966 generic.go:356] "Generic (PLEG): container finished" podID="ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" containerID="d342671d383ae933b3cd567123a35e35b7ca98bd35bdfcd571e1591d6dd3cd92" exitCode=0 Nov 02 20:12:45.203156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:45.203029 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" event={"ID":"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e","Type":"ContainerDied","Data":"d342671d383ae933b3cd567123a35e35b7ca98bd35bdfcd571e1591d6dd3cd92"} Nov 02 20:12:45.203156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:45.203080 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" event={"ID":"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e","Type":"ContainerStarted","Data":"1057c04d2e54b025b0df6cc47c91da5cddcfd24513c346fa0ab9fc8684dd0edd"} Nov 02 20:12:46.364495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.364468 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:46.506458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.506424 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-openshift-ca\") pod \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " Nov 02 20:12:46.506626 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.506477 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-kube-api-access-xzwbp\" (UniqueName: \"kubernetes.io/projected/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-kube-api-access-xzwbp\") pod \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " Nov 02 20:12:46.506626 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.506583 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-xks-ca\") pod \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\" (UID: \"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e\") " Nov 02 20:12:46.506859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.506830 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-openshift-ca" pod "ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" (UID: "ea9b3a2d-d165-48b5-b31c-5f33ab80d70e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:12:46.508662 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.508636 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-xks-ca" pod "ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" (UID: "ea9b3a2d-d165-48b5-b31c-5f33ab80d70e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:12:46.508781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.508707 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-kube-api-access-xzwbp" pod "ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" (UID: "ea9b3a2d-d165-48b5-b31c-5f33ab80d70e"). InnerVolumeSpecName "kube-api-access-xzwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:12:46.608321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.608259 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:46.608321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.608289 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwbp\" (UniqueName: \"kubernetes.io/projected/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-kube-api-access-xzwbp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:46.608321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:46.608328 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:12:47.217163 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:47.217128 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" Nov 02 20:12:47.217163 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:47.217153 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs" event={"ID":"ea9b3a2d-d165-48b5-b31c-5f33ab80d70e","Type":"ContainerDied","Data":"1057c04d2e54b025b0df6cc47c91da5cddcfd24513c346fa0ab9fc8684dd0edd"} Nov 02 20:12:47.217406 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:47.217206 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1057c04d2e54b025b0df6cc47c91da5cddcfd24513c346fa0ab9fc8684dd0edd" Nov 02 20:12:55.123352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:55.123314 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd"] Nov 02 20:12:55.231338 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:55.231286 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf"] Nov 02 20:12:58.838829 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:58.838793 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4"] Nov 02 20:12:58.839355 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:12:58.839000 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" containerID="cri-o://7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae" gracePeriod=30 Nov 02 20:13:00.123838 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.123769 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" podUID="1992fc16-5363-4bc9-a2a9-b6da4d39d63d" containerName="payload-processing" containerID="cri-o://38d8cfd5a6133bb754bf9391352584c45dcaaf90a60ce5559977cd4a56fc91eb" gracePeriod=25 Nov 02 20:13:00.231849 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.231810 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" podUID="19eabeab-8fb3-40db-9cfc-ae1512113297" containerName="payload-pre-processing" containerID="cri-o://61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a" gracePeriod=25 Nov 02 20:13:00.339421 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.339292 4966 generic.go:356] "Generic (PLEG): container finished" podID="1992fc16-5363-4bc9-a2a9-b6da4d39d63d" containerID="38d8cfd5a6133bb754bf9391352584c45dcaaf90a60ce5559977cd4a56fc91eb" exitCode=0 Nov 02 20:13:00.339775 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.339730 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" event={"ID":"1992fc16-5363-4bc9-a2a9-b6da4d39d63d","Type":"ContainerDied","Data":"38d8cfd5a6133bb754bf9391352584c45dcaaf90a60ce5559977cd4a56fc91eb"} Nov 02 20:13:00.424178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.424153 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:13:00.475457 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.475420 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-plugins-config-volume\") pod \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " Nov 02 20:13:00.475670 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.475652 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-kube-api-access-pphtr\" (UniqueName: \"kubernetes.io/projected/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-kube-api-access-pphtr\") pod \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\" (UID: \"1992fc16-5363-4bc9-a2a9-b6da4d39d63d\") " Nov 02 20:13:00.475938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.475910 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-plugins-config-volume" pod "1992fc16-5363-4bc9-a2a9-b6da4d39d63d" (UID: "1992fc16-5363-4bc9-a2a9-b6da4d39d63d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:13:00.476562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.476492 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:00.484737 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.484706 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-kube-api-access-pphtr" pod "1992fc16-5363-4bc9-a2a9-b6da4d39d63d" (UID: "1992fc16-5363-4bc9-a2a9-b6da4d39d63d"). InnerVolumeSpecName "kube-api-access-pphtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:00.515855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.515832 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:13:00.577364 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.577322 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19eabeab-8fb3-40db-9cfc-ae1512113297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eabeab-8fb3-40db-9cfc-ae1512113297-plugins-config-volume\") pod \"19eabeab-8fb3-40db-9cfc-ae1512113297\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " Nov 02 20:13:00.577558 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.577431 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19eabeab-8fb3-40db-9cfc-ae1512113297-kube-api-access-g9bll\" (UniqueName: \"kubernetes.io/projected/19eabeab-8fb3-40db-9cfc-ae1512113297-kube-api-access-g9bll\") pod \"19eabeab-8fb3-40db-9cfc-ae1512113297\" (UID: \"19eabeab-8fb3-40db-9cfc-ae1512113297\") " Nov 02 20:13:00.577767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.577741 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19eabeab-8fb3-40db-9cfc-ae1512113297-plugins-config-volume" pod "19eabeab-8fb3-40db-9cfc-ae1512113297" (UID: "19eabeab-8fb3-40db-9cfc-ae1512113297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:13:00.578011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.577991 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphtr\" (UniqueName: \"kubernetes.io/projected/1992fc16-5363-4bc9-a2a9-b6da4d39d63d-kube-api-access-pphtr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:00.578112 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.578016 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19eabeab-8fb3-40db-9cfc-ae1512113297-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:00.579965 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.579942 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19eabeab-8fb3-40db-9cfc-ae1512113297-kube-api-access-g9bll" pod "19eabeab-8fb3-40db-9cfc-ae1512113297" (UID: "19eabeab-8fb3-40db-9cfc-ae1512113297"). InnerVolumeSpecName "kube-api-access-g9bll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:00.679275 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:00.679180 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bll\" (UniqueName: \"kubernetes.io/projected/19eabeab-8fb3-40db-9cfc-ae1512113297-kube-api-access-g9bll\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:01.348714 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.348684 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" Nov 02 20:13:01.349158 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.348718 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd" event={"ID":"1992fc16-5363-4bc9-a2a9-b6da4d39d63d","Type":"ContainerDied","Data":"682e437efa06eaf3fb34be2bdb8995fa9bd4717fb6cf7bd25447cf5157b3829c"} Nov 02 20:13:01.349158 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.348779 4966 scope.go:122] "RemoveContainer" containerID="38d8cfd5a6133bb754bf9391352584c45dcaaf90a60ce5559977cd4a56fc91eb" Nov 02 20:13:01.350420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.350394 4966 generic.go:356] "Generic (PLEG): container finished" podID="19eabeab-8fb3-40db-9cfc-ae1512113297" containerID="61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a" exitCode=0 Nov 02 20:13:01.350531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.350433 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" Nov 02 20:13:01.350531 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.350492 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" event={"ID":"19eabeab-8fb3-40db-9cfc-ae1512113297","Type":"ContainerDied","Data":"61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a"} Nov 02 20:13:01.350617 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.350551 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf" event={"ID":"19eabeab-8fb3-40db-9cfc-ae1512113297","Type":"ContainerDied","Data":"896003a7a06fc26697132781ea4d7e2dec9e8a105256669dd3af343c6ac49204"} Nov 02 20:13:01.360661 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.360637 4966 scope.go:122] "RemoveContainer" containerID="61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a" Nov 02 20:13:01.371403 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.371383 4966 scope.go:122] "RemoveContainer" containerID="61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a" Nov 02 20:13:01.371677 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:13:01.371654 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a\": container with ID starting with 61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a not found: ID does not exist" containerID="61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a" Nov 02 20:13:01.371727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.371688 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a"} err="failed to get container status \"61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a\": rpc error: code = NotFound desc = could not find container \"61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a\": container with ID starting with 61217bf03854ddaf060fb144f690454f8763cb67f73a6c86ea5e049147cc246a not found: ID does not exist" Nov 02 20:13:01.500764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.500724 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd"] Nov 02 20:13:01.546532 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.546482 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b45a2b-77d8cb886f-gcghd"] Nov 02 20:13:01.646840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.646806 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf"] Nov 02 20:13:01.651764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:01.651729 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b45a2b-546cc8d659-p2tvf"] Nov 02 20:13:02.047941 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:02.047901 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:13:02.048136 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:02.047980 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:13:02.183667 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:02.183634 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1992fc16-5363-4bc9-a2a9-b6da4d39d63d" path="/var/lib/kubelet/pods/1992fc16-5363-4bc9-a2a9-b6da4d39d63d/volumes" Nov 02 20:13:02.184127 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:02.184108 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19eabeab-8fb3-40db-9cfc-ae1512113297" path="/var/lib/kubelet/pods/19eabeab-8fb3-40db-9cfc-ae1512113297/volumes" Nov 02 20:13:04.139943 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.139911 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 20:13:04.219731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.219699 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-credential-socket\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.219938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.219747 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb54a60-8aef-48f1-893c-daeca2a1f550-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb54a60-8aef-48f1-893c-daeca2a1f550-istiod-ca-cert\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.219938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.219767 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-data\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.219938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.219799 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-podinfo\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.219938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.219863 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-kube-api-access-cz9z9\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-kube-api-access-cz9z9\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.220160 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220024 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-socket\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.220160 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220049 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-credential-socket" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:04.220160 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220085 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-envoy\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.220342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220163 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-certs\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.220342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220200 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-data" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:04.220342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220220 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb54a60-8aef-48f1-893c-daeca2a1f550-istiod-ca-cert" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:13:04.220342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220239 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-token\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-token\") pod \"ecb54a60-8aef-48f1-893c-daeca2a1f550\" (UID: \"ecb54a60-8aef-48f1-893c-daeca2a1f550\") " Nov 02 20:13:04.220614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220591 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-socket" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:04.220765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220608 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-certs" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:04.220873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220815 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.220978 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.220963 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb54a60-8aef-48f1-893c-daeca2a1f550-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.221081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.221067 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.222572 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.222540 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-envoy" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:04.222960 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.222939 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-kube-api-access-cz9z9" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "kube-api-access-cz9z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:04.222960 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.222950 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-podinfo" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:13:04.223132 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.223110 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-token" pod "ecb54a60-8aef-48f1-893c-daeca2a1f550" (UID: "ecb54a60-8aef-48f1-893c-daeca2a1f550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:04.322445 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.322406 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.322445 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.322445 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.322445 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.322460 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.322776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.322474 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb54a60-8aef-48f1-893c-daeca2a1f550-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.322776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.322489 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9z9\" (UniqueName: \"kubernetes.io/projected/ecb54a60-8aef-48f1-893c-daeca2a1f550-kube-api-access-cz9z9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.322776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.322504 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb54a60-8aef-48f1-893c-daeca2a1f550-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:04.374802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.374718 4966 generic.go:356] "Generic (PLEG): container finished" podID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerID="7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae" exitCode=0 Nov 02 20:13:04.375025 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.374804 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" event={"ID":"ecb54a60-8aef-48f1-893c-daeca2a1f550","Type":"ContainerDied","Data":"7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae"} Nov 02 20:13:04.375210 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.374858 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" Nov 02 20:13:04.375336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.375189 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4" event={"ID":"ecb54a60-8aef-48f1-893c-daeca2a1f550","Type":"ContainerDied","Data":"71530e69da5f22ae658f71382f0645298d500f95e65c0b35209213824ecfd8b2"} Nov 02 20:13:04.375422 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.375290 4966 scope.go:122] "RemoveContainer" containerID="7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae" Nov 02 20:13:04.392827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.392802 4966 scope.go:122] "RemoveContainer" containerID="7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae" Nov 02 20:13:04.393322 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:13:04.393206 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae\": container with ID starting with 7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae not found: ID does not exist" containerID="7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae" Nov 02 20:13:04.393322 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.393248 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae"} err="failed to get container status \"7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae\": rpc error: code = NotFound desc = could not find container \"7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae\": container with ID starting with 7fa4613bba70a2032aa5cc8cf47e72537015d651adf1192eb4638ebeb67f83ae not found: ID does not exist" Nov 02 20:13:04.531773 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.530847 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4"] Nov 02 20:13:04.534138 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:04.534111 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f5bh4"] Nov 02 20:13:06.186007 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:06.185969 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" path="/var/lib/kubelet/pods/ecb54a60-8aef-48f1-893c-daeca2a1f550/volumes" Nov 02 20:13:14.753900 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:13:14.753856 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:13:15.042268 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:15.042188 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:13:15.058158 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:15.058118 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:13:15.062825 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:15.062797 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:13:15.078115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:15.078087 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:13:19.025745 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:19.025711 4966 scope.go:122] "RemoveContainer" containerID="a5b781d29a8716d847842b2fcd9ec369970ac5a66628511498b9f8c787fb28f5" Nov 02 20:13:19.039057 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:19.039033 4966 scope.go:122] "RemoveContainer" containerID="3c3b1e6da060885fcea4674ad29b0fc7a51e684503a7b2dde87ebb6d2b94cbba" Nov 02 20:13:28.692848 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.692807 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj"] Nov 02 20:13:28.693367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693352 4966 cpu_manager.go:414] "containerMap: removing container" podUID="1992fc16-5363-4bc9-a2a9-b6da4d39d63d" containerName="payload-processing" Nov 02 20:13:28.693424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693368 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1992fc16-5363-4bc9-a2a9-b6da4d39d63d" containerName="payload-processing" Nov 02 20:13:28.693424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693390 4966 cpu_manager.go:414] "containerMap: removing container" podUID="19eabeab-8fb3-40db-9cfc-ae1512113297" containerName="payload-pre-processing" Nov 02 20:13:28.693424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693396 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19eabeab-8fb3-40db-9cfc-ae1512113297" containerName="payload-pre-processing" Nov 02 20:13:28.693424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693412 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" Nov 02 20:13:28.693424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693417 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" Nov 02 20:13:28.693594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693427 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" containerName="revoke-keys" Nov 02 20:13:28.693594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693433 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" containerName="revoke-keys" Nov 02 20:13:28.693594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693511 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" containerName="revoke-keys" Nov 02 20:13:28.693594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693528 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1992fc16-5363-4bc9-a2a9-b6da4d39d63d" containerName="payload-processing" Nov 02 20:13:28.693594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693535 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="19eabeab-8fb3-40db-9cfc-ae1512113297" containerName="payload-pre-processing" Nov 02 20:13:28.693594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.693541 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb54a60-8aef-48f1-893c-daeca2a1f550" containerName="istio-proxy" Nov 02 20:13:28.698471 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.698452 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.751180 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.751143 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj"] Nov 02 20:13:28.792187 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.792134 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:13:28.792187 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.792142 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:13:28.793658 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.793632 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:13:28.793658 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.793644 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:13:28.810115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.810085 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.810266 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.810141 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.810266 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.810212 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5j5\" (UniqueName: \"kubernetes.io/projected/83f85457-8e3c-4845-962b-da05ca3822f5-kube-api-access-gt5j5\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.911366 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.911327 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.911567 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.911384 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.911567 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.911417 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5j5\" (UniqueName: \"kubernetes.io/projected/83f85457-8e3c-4845-962b-da05ca3822f5-kube-api-access-gt5j5\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.912052 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.912024 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.913643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.913623 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:28.965460 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:28.965385 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5j5\" (UniqueName: \"kubernetes.io/projected/83f85457-8e3c-4845-962b-da05ca3822f5-kube-api-access-gt5j5\") pod \"maas-api-revoke-sub-01b9d465-vwnsj\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:29.012741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:29.012693 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:29.392838 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:29.392800 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj"] Nov 02 20:13:29.393370 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:13:29.393339 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f85457_8e3c_4845_962b_da05ca3822f5.slice/crio-d0395aae8f8ff3d371d6ae4cc7585228ffc7d0f8dae72c1f23f28b8a4fbaf4d1 WatchSource:0}: Error finding container d0395aae8f8ff3d371d6ae4cc7585228ffc7d0f8dae72c1f23f28b8a4fbaf4d1: Status 404 returned error can't find the container with id d0395aae8f8ff3d371d6ae4cc7585228ffc7d0f8dae72c1f23f28b8a4fbaf4d1 Nov 02 20:13:29.548700 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:29.548658 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" event={"ID":"83f85457-8e3c-4845-962b-da05ca3822f5","Type":"ContainerStarted","Data":"d0395aae8f8ff3d371d6ae4cc7585228ffc7d0f8dae72c1f23f28b8a4fbaf4d1"} Nov 02 20:13:30.556191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:30.556153 4966 generic.go:356] "Generic (PLEG): container finished" podID="83f85457-8e3c-4845-962b-da05ca3822f5" containerID="c7d907f3034418c7e14096b4c84832412d68b201c20e3b9d7cabf9d684f4a81f" exitCode=0 Nov 02 20:13:30.556645 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:30.556247 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" event={"ID":"83f85457-8e3c-4845-962b-da05ca3822f5","Type":"ContainerDied","Data":"c7d907f3034418c7e14096b4c84832412d68b201c20e3b9d7cabf9d684f4a81f"} Nov 02 20:13:31.720057 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.720032 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:31.846138 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.846060 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-xks-ca\") pod \"83f85457-8e3c-4845-962b-da05ca3822f5\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " Nov 02 20:13:31.846138 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.846122 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-openshift-ca\") pod \"83f85457-8e3c-4845-962b-da05ca3822f5\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " Nov 02 20:13:31.846356 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.846152 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f85457-8e3c-4845-962b-da05ca3822f5-kube-api-access-gt5j5\" (UniqueName: \"kubernetes.io/projected/83f85457-8e3c-4845-962b-da05ca3822f5-kube-api-access-gt5j5\") pod \"83f85457-8e3c-4845-962b-da05ca3822f5\" (UID: \"83f85457-8e3c-4845-962b-da05ca3822f5\") " Nov 02 20:13:31.846599 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.846568 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-openshift-ca" pod "83f85457-8e3c-4845-962b-da05ca3822f5" (UID: "83f85457-8e3c-4845-962b-da05ca3822f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:13:31.848463 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.848437 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f85457-8e3c-4845-962b-da05ca3822f5-kube-api-access-gt5j5" pod "83f85457-8e3c-4845-962b-da05ca3822f5" (UID: "83f85457-8e3c-4845-962b-da05ca3822f5"). InnerVolumeSpecName "kube-api-access-gt5j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:31.848759 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.848731 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-xks-ca" pod "83f85457-8e3c-4845-962b-da05ca3822f5" (UID: "83f85457-8e3c-4845-962b-da05ca3822f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:13:31.947590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.947550 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:31.947590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.947583 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f85457-8e3c-4845-962b-da05ca3822f5-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:31.947590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:31.947594 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5j5\" (UniqueName: \"kubernetes.io/projected/83f85457-8e3c-4845-962b-da05ca3822f5-kube-api-access-gt5j5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:32.570156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:32.570126 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" Nov 02 20:13:32.570379 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:32.570121 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj" event={"ID":"83f85457-8e3c-4845-962b-da05ca3822f5","Type":"ContainerDied","Data":"d0395aae8f8ff3d371d6ae4cc7585228ffc7d0f8dae72c1f23f28b8a4fbaf4d1"} Nov 02 20:13:32.570379 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:32.570287 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0395aae8f8ff3d371d6ae4cc7585228ffc7d0f8dae72c1f23f28b8a4fbaf4d1" Nov 02 20:13:40.530799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:40.530759 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb"] Nov 02 20:13:41.465546 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.465499 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft"] Nov 02 20:13:41.466099 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.466083 4966 cpu_manager.go:414] "containerMap: removing container" podUID="83f85457-8e3c-4845-962b-da05ca3822f5" containerName="revoke-keys" Nov 02 20:13:41.466156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.466101 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f85457-8e3c-4845-962b-da05ca3822f5" containerName="revoke-keys" Nov 02 20:13:41.466210 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.466201 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f85457-8e3c-4845-962b-da05ca3822f5" containerName="revoke-keys" Nov 02 20:13:41.470154 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.470134 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.554899 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.554861 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.555287 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.554923 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.555287 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.555053 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5mf\" (UniqueName: \"kubernetes.io/projected/621b4a1e-76f1-4766-9429-fafd758c4d0b-kube-api-access-pz5mf\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.565573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.565539 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:13:41.565573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.565554 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:13:41.565573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.565540 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:13:41.565773 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.565618 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:13:41.569027 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.569006 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft"] Nov 02 20:13:41.656796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.656763 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.656997 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.656820 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.656997 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.656855 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5mf\" (UniqueName: \"kubernetes.io/projected/621b4a1e-76f1-4766-9429-fafd758c4d0b-kube-api-access-pz5mf\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.657623 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.657594 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.659453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.659427 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.709513 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.709481 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5mf\" (UniqueName: \"kubernetes.io/projected/621b4a1e-76f1-4766-9429-fafd758c4d0b-kube-api-access-pz5mf\") pod \"maas-api-revoke-sub-01b751c2-28kft\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:41.787453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:41.787416 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:42.161234 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:42.161201 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft"] Nov 02 20:13:42.161410 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:13:42.161269 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621b4a1e_76f1_4766_9429_fafd758c4d0b.slice/crio-87395f0fa154b4b18a09433d9314d067ec236c39d7ab853565ea9f9f401e35f5 WatchSource:0}: Error finding container 87395f0fa154b4b18a09433d9314d067ec236c39d7ab853565ea9f9f401e35f5: Status 404 returned error can't find the container with id 87395f0fa154b4b18a09433d9314d067ec236c39d7ab853565ea9f9f401e35f5 Nov 02 20:13:42.640189 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:42.640141 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" event={"ID":"621b4a1e-76f1-4766-9429-fafd758c4d0b","Type":"ContainerStarted","Data":"388b6c15d0a74dc74fc4d7dc6bbedeb9181f52fbd63617ee0b307b8b51195e8c"} Nov 02 20:13:42.640722 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:42.640222 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" event={"ID":"621b4a1e-76f1-4766-9429-fafd758c4d0b","Type":"ContainerStarted","Data":"87395f0fa154b4b18a09433d9314d067ec236c39d7ab853565ea9f9f401e35f5"} Nov 02 20:13:42.747201 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:42.747145 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" podStartSLOduration=1.747130012 podStartE2EDuration="1.747130012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:42.744522747 +0000 UTC m=+2728.693178585" watchObservedRunningTime="2026-11-02 20:13:42.747130012 +0000 UTC m=+2728.695785895" Nov 02 20:13:43.648147 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:43.648115 4966 generic.go:356] "Generic (PLEG): container finished" podID="621b4a1e-76f1-4766-9429-fafd758c4d0b" containerID="388b6c15d0a74dc74fc4d7dc6bbedeb9181f52fbd63617ee0b307b8b51195e8c" exitCode=0 Nov 02 20:13:43.648594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:43.648158 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" event={"ID":"621b4a1e-76f1-4766-9429-fafd758c4d0b","Type":"ContainerDied","Data":"388b6c15d0a74dc74fc4d7dc6bbedeb9181f52fbd63617ee0b307b8b51195e8c"} Nov 02 20:13:44.765030 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:13:44.764994 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:13:44.807549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.807519 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:44.898699 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.898661 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-openshift-ca\") pod \"621b4a1e-76f1-4766-9429-fafd758c4d0b\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " Nov 02 20:13:44.898897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.898853 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-xks-ca\") pod \"621b4a1e-76f1-4766-9429-fafd758c4d0b\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " Nov 02 20:13:44.898965 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.898897 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621b4a1e-76f1-4766-9429-fafd758c4d0b-kube-api-access-pz5mf\" (UniqueName: \"kubernetes.io/projected/621b4a1e-76f1-4766-9429-fafd758c4d0b-kube-api-access-pz5mf\") pod \"621b4a1e-76f1-4766-9429-fafd758c4d0b\" (UID: \"621b4a1e-76f1-4766-9429-fafd758c4d0b\") " Nov 02 20:13:44.899036 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.899013 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-openshift-ca" pod "621b4a1e-76f1-4766-9429-fafd758c4d0b" (UID: "621b4a1e-76f1-4766-9429-fafd758c4d0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:13:44.899316 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.899285 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:44.900972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.900946 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-xks-ca" pod "621b4a1e-76f1-4766-9429-fafd758c4d0b" (UID: "621b4a1e-76f1-4766-9429-fafd758c4d0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:13:44.901091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:44.900983 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621b4a1e-76f1-4766-9429-fafd758c4d0b-kube-api-access-pz5mf" pod "621b4a1e-76f1-4766-9429-fafd758c4d0b" (UID: "621b4a1e-76f1-4766-9429-fafd758c4d0b"). InnerVolumeSpecName "kube-api-access-pz5mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:45.000972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:45.000924 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621b4a1e-76f1-4766-9429-fafd758c4d0b-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:45.000972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:45.000967 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5mf\" (UniqueName: \"kubernetes.io/projected/621b4a1e-76f1-4766-9429-fafd758c4d0b-kube-api-access-pz5mf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:45.665869 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:45.665839 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" Nov 02 20:13:45.665869 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:45.665855 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft" event={"ID":"621b4a1e-76f1-4766-9429-fafd758c4d0b","Type":"ContainerDied","Data":"87395f0fa154b4b18a09433d9314d067ec236c39d7ab853565ea9f9f401e35f5"} Nov 02 20:13:45.666149 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:45.665903 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87395f0fa154b4b18a09433d9314d067ec236c39d7ab853565ea9f9f401e35f5" Nov 02 20:13:55.552413 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.552293 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" podUID="b1c70c08-8340-44eb-a9cb-711cc360cb3d" containerName="main" containerID="cri-o://63712d675863aeaf2ce2b0eaf411b190067c01a97f2b6a10c26e392e792f577e" gracePeriod=45 Nov 02 20:13:55.744175 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.743924 4966 generic.go:356] "Generic (PLEG): container finished" podID="b1c70c08-8340-44eb-a9cb-711cc360cb3d" containerID="63712d675863aeaf2ce2b0eaf411b190067c01a97f2b6a10c26e392e792f577e" exitCode=0 Nov 02 20:13:55.744175 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.744126 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" event={"ID":"b1c70c08-8340-44eb-a9cb-711cc360cb3d","Type":"ContainerDied","Data":"63712d675863aeaf2ce2b0eaf411b190067c01a97f2b6a10c26e392e792f577e"} Nov 02 20:13:55.827899 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.827821 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:13:55.923617 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.923582 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tls-certs\") pod \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " Nov 02 20:13:55.923808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.923636 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-model-cache\") pod \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " Nov 02 20:13:55.923808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.923689 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tmp-dir\") pod \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " Nov 02 20:13:55.923808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.923709 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c70c08-8340-44eb-a9cb-711cc360cb3d-kube-api-access-7fdk2\" (UniqueName: \"kubernetes.io/projected/b1c70c08-8340-44eb-a9cb-711cc360cb3d-kube-api-access-7fdk2\") pod \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " Nov 02 20:13:55.923997 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.923807 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-home\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-home\") pod \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " Nov 02 20:13:55.923997 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.923878 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-dshm\") pod \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\" (UID: \"b1c70c08-8340-44eb-a9cb-711cc360cb3d\") " Nov 02 20:13:55.924106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.923995 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-model-cache" pod "b1c70c08-8340-44eb-a9cb-711cc360cb3d" (UID: "b1c70c08-8340-44eb-a9cb-711cc360cb3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:55.924106 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.924010 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tmp-dir" pod "b1c70c08-8340-44eb-a9cb-711cc360cb3d" (UID: "b1c70c08-8340-44eb-a9cb-711cc360cb3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:55.924229 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.924201 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-home" pod "b1c70c08-8340-44eb-a9cb-711cc360cb3d" (UID: "b1c70c08-8340-44eb-a9cb-711cc360cb3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:55.924438 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.924417 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:55.924535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.924441 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:55.924535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.924457 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:55.926375 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.926348 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tls-certs" pod "b1c70c08-8340-44eb-a9cb-711cc360cb3d" (UID: "b1c70c08-8340-44eb-a9cb-711cc360cb3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:13:55.926486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.926405 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-dshm" pod "b1c70c08-8340-44eb-a9cb-711cc360cb3d" (UID: "b1c70c08-8340-44eb-a9cb-711cc360cb3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:13:55.926486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:55.926468 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c70c08-8340-44eb-a9cb-711cc360cb3d-kube-api-access-7fdk2" pod "b1c70c08-8340-44eb-a9cb-711cc360cb3d" (UID: "b1c70c08-8340-44eb-a9cb-711cc360cb3d"). InnerVolumeSpecName "kube-api-access-7fdk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:13:56.026135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.026090 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c70c08-8340-44eb-a9cb-711cc360cb3d-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:56.026135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.026132 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c70c08-8340-44eb-a9cb-711cc360cb3d-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:56.026135 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.026147 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fdk2\" (UniqueName: \"kubernetes.io/projected/b1c70c08-8340-44eb-a9cb-711cc360cb3d-kube-api-access-7fdk2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:13:56.754540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.754506 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" Nov 02 20:13:56.754540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.754526 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb" event={"ID":"b1c70c08-8340-44eb-a9cb-711cc360cb3d","Type":"ContainerDied","Data":"a6c156e53a5c4f8570ea7a3ee5a345cf78d49db203d701978e8374d475ee322f"} Nov 02 20:13:56.755085 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.754578 4966 scope.go:122] "RemoveContainer" containerID="63712d675863aeaf2ce2b0eaf411b190067c01a97f2b6a10c26e392e792f577e" Nov 02 20:13:56.908857 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.908821 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb"] Nov 02 20:13:56.954569 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:56.954488 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-838bc1/test-model-838bc1-kserve-7f4556c7cc-npntb"] Nov 02 20:13:58.183735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.183693 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c70c08-8340-44eb-a9cb-711cc360cb3d" path="/var/lib/kubelet/pods/b1c70c08-8340-44eb-a9cb-711cc360cb3d/volumes" Nov 02 20:13:58.423364 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.423325 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb"] Nov 02 20:13:58.423873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.423859 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b1c70c08-8340-44eb-a9cb-711cc360cb3d" containerName="main" Nov 02 20:13:58.423923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.423874 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c70c08-8340-44eb-a9cb-711cc360cb3d" containerName="main" Nov 02 20:13:58.423923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.423895 4966 cpu_manager.go:414] "containerMap: removing container" podUID="621b4a1e-76f1-4766-9429-fafd758c4d0b" containerName="revoke-keys" Nov 02 20:13:58.423923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.423900 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621b4a1e-76f1-4766-9429-fafd758c4d0b" containerName="revoke-keys" Nov 02 20:13:58.424032 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.423984 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c70c08-8340-44eb-a9cb-711cc360cb3d" containerName="main" Nov 02 20:13:58.424032 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.423995 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="621b4a1e-76f1-4766-9429-fafd758c4d0b" containerName="revoke-keys" Nov 02 20:13:58.428638 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.428616 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.455747 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.455712 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.455747 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.455743 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccvr\" (UniqueName: \"kubernetes.io/projected/f5056aa2-c2e7-4fd4-8463-51d901ae282f-kube-api-access-cccvr\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.455979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.455868 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.481041 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.481007 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb"] Nov 02 20:13:58.522546 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.522511 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:13:58.522768 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.522744 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:13:58.522832 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.522818 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:13:58.523696 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.523679 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:13:58.556630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.556586 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.556758 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.556632 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccvr\" (UniqueName: \"kubernetes.io/projected/f5056aa2-c2e7-4fd4-8463-51d901ae282f-kube-api-access-cccvr\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.556758 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.556708 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.557293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.557273 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.559289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.559265 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.609550 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.609527 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccvr\" (UniqueName: \"kubernetes.io/projected/f5056aa2-c2e7-4fd4-8463-51d901ae282f-kube-api-access-cccvr\") pod \"maas-api-revoke-sub-14a67b71-mf6lb\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:58.742416 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:58.742380 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:13:59.117291 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:59.117260 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb"] Nov 02 20:13:59.118620 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:13:59.118591 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5056aa2_c2e7_4fd4_8463_51d901ae282f.slice/crio-288f7ca57dae8668a4426f778f1e6398fbb165c9e15b83dcdf2cf3c6c208e349 WatchSource:0}: Error finding container 288f7ca57dae8668a4426f778f1e6398fbb165c9e15b83dcdf2cf3c6c208e349: Status 404 returned error can't find the container with id 288f7ca57dae8668a4426f778f1e6398fbb165c9e15b83dcdf2cf3c6c208e349 Nov 02 20:13:59.782678 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:59.782639 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" event={"ID":"f5056aa2-c2e7-4fd4-8463-51d901ae282f","Type":"ContainerStarted","Data":"48edfc0b3e4173adba0a484302af063aa2491ff0d431b9e07976e9ccdb0b7e0c"} Nov 02 20:13:59.783172 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:59.782707 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" event={"ID":"f5056aa2-c2e7-4fd4-8463-51d901ae282f","Type":"ContainerStarted","Data":"288f7ca57dae8668a4426f778f1e6398fbb165c9e15b83dcdf2cf3c6c208e349"} Nov 02 20:13:59.889362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:13:59.889287 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" podStartSLOduration=1.889272172 podStartE2EDuration="1.889272172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:59.886635752 +0000 UTC m=+2745.835291590" watchObservedRunningTime="2026-11-02 20:13:59.889272172 +0000 UTC m=+2745.837928011" Nov 02 20:14:00.792481 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:00.792441 4966 generic.go:356] "Generic (PLEG): container finished" podID="f5056aa2-c2e7-4fd4-8463-51d901ae282f" containerID="48edfc0b3e4173adba0a484302af063aa2491ff0d431b9e07976e9ccdb0b7e0c" exitCode=0 Nov 02 20:14:00.793020 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:00.792529 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" event={"ID":"f5056aa2-c2e7-4fd4-8463-51d901ae282f","Type":"ContainerDied","Data":"48edfc0b3e4173adba0a484302af063aa2491ff0d431b9e07976e9ccdb0b7e0c"} Nov 02 20:14:01.946385 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:01.946354 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:14:01.997691 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:01.997652 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5056aa2-c2e7-4fd4-8463-51d901ae282f-kube-api-access-cccvr\" (UniqueName: \"kubernetes.io/projected/f5056aa2-c2e7-4fd4-8463-51d901ae282f-kube-api-access-cccvr\") pod \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " Nov 02 20:14:01.997921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:01.997753 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-openshift-ca\") pod \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " Nov 02 20:14:01.998008 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:01.997946 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-xks-ca\") pod \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\" (UID: \"f5056aa2-c2e7-4fd4-8463-51d901ae282f\") " Nov 02 20:14:01.998202 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:01.998160 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-openshift-ca" pod "f5056aa2-c2e7-4fd4-8463-51d901ae282f" (UID: "f5056aa2-c2e7-4fd4-8463-51d901ae282f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:01.998417 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:01.998395 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:02.000120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:02.000086 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-xks-ca" pod "f5056aa2-c2e7-4fd4-8463-51d901ae282f" (UID: "f5056aa2-c2e7-4fd4-8463-51d901ae282f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:02.000251 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:02.000234 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5056aa2-c2e7-4fd4-8463-51d901ae282f-kube-api-access-cccvr" pod "f5056aa2-c2e7-4fd4-8463-51d901ae282f" (UID: "f5056aa2-c2e7-4fd4-8463-51d901ae282f"). InnerVolumeSpecName "kube-api-access-cccvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:02.102549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:02.102451 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5056aa2-c2e7-4fd4-8463-51d901ae282f-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:02.102549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:02.102485 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cccvr\" (UniqueName: \"kubernetes.io/projected/f5056aa2-c2e7-4fd4-8463-51d901ae282f-kube-api-access-cccvr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:02.807400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:02.807361 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" Nov 02 20:14:02.807400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:02.807388 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb" event={"ID":"f5056aa2-c2e7-4fd4-8463-51d901ae282f","Type":"ContainerDied","Data":"288f7ca57dae8668a4426f778f1e6398fbb165c9e15b83dcdf2cf3c6c208e349"} Nov 02 20:14:02.807669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:02.807442 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288f7ca57dae8668a4426f778f1e6398fbb165c9e15b83dcdf2cf3c6c208e349" Nov 02 20:14:14.274840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:14.274807 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh"] Nov 02 20:14:14.275370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:14.275068 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" containerID="cri-o://da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b" gracePeriod=30 Nov 02 20:14:14.752816 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:14:14.752780 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:14:19.589289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.589250 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 20:14:19.701918 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.701819 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-socket\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.701918 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.701878 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-data\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702172 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.701936 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-kube-api-access-w9z89\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-kube-api-access-w9z89\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702172 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.701964 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-envoy\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702172 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.701997 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-token\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-token\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702172 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702096 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-podinfo\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702172 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702135 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de86340-39ff-40bf-91f4-4dd3e8136c12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de86340-39ff-40bf-91f4-4dd3e8136c12-istiod-ca-cert\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702172 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702145 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-socket" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.702552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702181 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-data" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.702552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702212 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-credential-socket\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702254 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-certs\") pod \"3de86340-39ff-40bf-91f4-4dd3e8136c12\" (UID: \"3de86340-39ff-40bf-91f4-4dd3e8136c12\") " Nov 02 20:14:19.702710 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702620 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-credential-socket" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.702710 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702627 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de86340-39ff-40bf-91f4-4dd3e8136c12-istiod-ca-cert" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:19.702826 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702772 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-certs" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.702890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702836 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de86340-39ff-40bf-91f4-4dd3e8136c12-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.702890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702858 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.702890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702872 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.702890 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.702885 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.704678 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.704653 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-envoy" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:19.705008 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.704978 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-podinfo" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:14:19.705120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.705097 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-kube-api-access-w9z89" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "kube-api-access-w9z89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:19.705169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.705125 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-token" pod "3de86340-39ff-40bf-91f4-4dd3e8136c12" (UID: "3de86340-39ff-40bf-91f4-4dd3e8136c12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:19.804584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.804547 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9z89\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-kube-api-access-w9z89\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.804584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.804582 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.804861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.804595 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.804861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.804607 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de86340-39ff-40bf-91f4-4dd3e8136c12-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.804861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.804620 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de86340-39ff-40bf-91f4-4dd3e8136c12-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.927125 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.927087 4966 generic.go:356] "Generic (PLEG): container finished" podID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerID="da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b" exitCode=0 Nov 02 20:14:19.927312 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.927129 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" Nov 02 20:14:19.927312 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.927164 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" event={"ID":"3de86340-39ff-40bf-91f4-4dd3e8136c12","Type":"ContainerDied","Data":"da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b"} Nov 02 20:14:19.927312 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.927218 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh" event={"ID":"3de86340-39ff-40bf-91f4-4dd3e8136c12","Type":"ContainerDied","Data":"6fb67f34d1f46ab36c969e461e3fde948ae1c503255288b0b317228cd35c01db"} Nov 02 20:14:19.927312 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.927267 4966 scope.go:122] "RemoveContainer" containerID="da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b" Nov 02 20:14:19.942013 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.941263 4966 scope.go:122] "RemoveContainer" containerID="da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b" Nov 02 20:14:19.942013 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:14:19.941632 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b\": container with ID starting with da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b not found: ID does not exist" containerID="da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b" Nov 02 20:14:19.942013 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:19.941665 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b"} err="failed to get container status \"da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b\": rpc error: code = NotFound desc = could not find container \"da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b\": container with ID starting with da7a03b6895660dea6634c2e30a871306151c422ce96c40012796caad72f1f4b not found: ID does not exist" Nov 02 20:14:20.082261 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:20.082221 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh"] Nov 02 20:14:20.089107 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:20.089076 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e9c144-openshift-default-58f654d4b5-hccrh"] Nov 02 20:14:20.187071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:20.187034 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" path="/var/lib/kubelet/pods/3de86340-39ff-40bf-91f4-4dd3e8136c12/volumes" Nov 02 20:14:21.304116 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.304072 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw"] Nov 02 20:14:21.860442 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.860400 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8"] Nov 02 20:14:21.861250 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.861222 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f5056aa2-c2e7-4fd4-8463-51d901ae282f" containerName="revoke-keys" Nov 02 20:14:21.861390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.861251 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5056aa2-c2e7-4fd4-8463-51d901ae282f" containerName="revoke-keys" Nov 02 20:14:21.861390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.861326 4966 cpu_manager.go:414] "containerMap: removing container" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" Nov 02 20:14:21.861390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.861337 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" Nov 02 20:14:21.861569 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.861496 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5056aa2-c2e7-4fd4-8463-51d901ae282f" containerName="revoke-keys" Nov 02 20:14:21.861569 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.861517 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de86340-39ff-40bf-91f4-4dd3e8136c12" containerName="istio-proxy" Nov 02 20:14:21.866960 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.866931 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:21.915320 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.915211 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:14:21.915517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.915377 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:14:21.919077 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.917184 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8"] Nov 02 20:14:21.926144 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.926115 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:21.926286 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.926267 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:21.926420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.926324 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tbx\" (UniqueName: \"kubernetes.io/projected/e332bca4-b357-4d7d-ba52-6563ce716794-kube-api-access-h7tbx\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:21.958628 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.958600 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:21.958767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:21.958667 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:14:22.027352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.027313 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:22.027563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.027369 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tbx\" (UniqueName: \"kubernetes.io/projected/e332bca4-b357-4d7d-ba52-6563ce716794-kube-api-access-h7tbx\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:22.027563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.027416 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:22.028512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.028483 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:22.030638 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.030609 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:22.080197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.080154 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tbx\" (UniqueName: \"kubernetes.io/projected/e332bca4-b357-4d7d-ba52-6563ce716794-kube-api-access-h7tbx\") pod \"maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:22.181909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.181759 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:22.567884 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.567750 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8"] Nov 02 20:14:22.570526 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:14:22.570494 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode332bca4_b357_4d7d_ba52_6563ce716794.slice/crio-346daf931af419024efbca43f5d780dbf2a0d4308a5db067f68ed8f45c26783b WatchSource:0}: Error finding container 346daf931af419024efbca43f5d780dbf2a0d4308a5db067f68ed8f45c26783b: Status 404 returned error can't find the container with id 346daf931af419024efbca43f5d780dbf2a0d4308a5db067f68ed8f45c26783b Nov 02 20:14:22.954466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:22.954427 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" event={"ID":"e332bca4-b357-4d7d-ba52-6563ce716794","Type":"ContainerStarted","Data":"346daf931af419024efbca43f5d780dbf2a0d4308a5db067f68ed8f45c26783b"} Nov 02 20:14:23.854142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.854095 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h"] Nov 02 20:14:23.858204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.858177 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:23.914812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.914739 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h"] Nov 02 20:14:23.949025 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.948993 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce845758-965b-496e-94a1-35908a53459d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:23.949178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.949037 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce845758-965b-496e-94a1-35908a53459d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:23.949178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.949073 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbdq\" (UniqueName: \"kubernetes.io/projected/ce845758-965b-496e-94a1-35908a53459d-kube-api-access-zgbdq\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:23.963581 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.963529 4966 generic.go:356] "Generic (PLEG): container finished" podID="e332bca4-b357-4d7d-ba52-6563ce716794" containerID="b0f87465fe0c834fe76bbc9e1b45d0f63566d48bf1dcc55eed88c383532ba4e0" exitCode=0 Nov 02 20:14:23.963742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:23.963614 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" event={"ID":"e332bca4-b357-4d7d-ba52-6563ce716794","Type":"ContainerDied","Data":"b0f87465fe0c834fe76bbc9e1b45d0f63566d48bf1dcc55eed88c383532ba4e0"} Nov 02 20:14:24.050077 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.050041 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce845758-965b-496e-94a1-35908a53459d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:24.050310 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.050091 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbdq\" (UniqueName: \"kubernetes.io/projected/ce845758-965b-496e-94a1-35908a53459d-kube-api-access-zgbdq\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:24.050310 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.050240 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce845758-965b-496e-94a1-35908a53459d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:24.050923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.050898 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce845758-965b-496e-94a1-35908a53459d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:24.052664 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.052639 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce845758-965b-496e-94a1-35908a53459d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:24.102752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.102719 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbdq\" (UniqueName: \"kubernetes.io/projected/ce845758-965b-496e-94a1-35908a53459d-kube-api-access-zgbdq\") pod \"maas-api-revoke-sub-70e5aed6-lvj9h\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:24.172264 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.172187 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:24.548058 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.548029 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h"] Nov 02 20:14:24.550228 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:14:24.550194 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce845758_965b_496e_94a1_35908a53459d.slice/crio-a0f21bde8fa5fe015f03f92b05cf4c4e82a0175f4ae5fe2f8bcb9fd5b541d204 WatchSource:0}: Error finding container a0f21bde8fa5fe015f03f92b05cf4c4e82a0175f4ae5fe2f8bcb9fd5b541d204: Status 404 returned error can't find the container with id a0f21bde8fa5fe015f03f92b05cf4c4e82a0175f4ae5fe2f8bcb9fd5b541d204 Nov 02 20:14:24.972193 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.972149 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" event={"ID":"ce845758-965b-496e-94a1-35908a53459d","Type":"ContainerStarted","Data":"0b82f4c553c987f2b052d4f4fcc4260bd655f9c708f6cc952990c729d4acf5b9"} Nov 02 20:14:24.972665 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:24.972207 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" event={"ID":"ce845758-965b-496e-94a1-35908a53459d","Type":"ContainerStarted","Data":"a0f21bde8fa5fe015f03f92b05cf4c4e82a0175f4ae5fe2f8bcb9fd5b541d204"} Nov 02 20:14:25.075074 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.074985 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" podStartSLOduration=2.074966239 podStartE2EDuration="2.074966239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:25.074368017 +0000 UTC m=+2771.023023856" watchObservedRunningTime="2026-11-02 20:14:25.074966239 +0000 UTC m=+2771.023622079" Nov 02 20:14:25.147164 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.147134 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:25.165072 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.165035 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e332bca4-b357-4d7d-ba52-6563ce716794-kube-api-access-h7tbx\" (UniqueName: \"kubernetes.io/projected/e332bca4-b357-4d7d-ba52-6563ce716794-kube-api-access-h7tbx\") pod \"e332bca4-b357-4d7d-ba52-6563ce716794\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " Nov 02 20:14:25.165206 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.165105 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-openshift-ca\") pod \"e332bca4-b357-4d7d-ba52-6563ce716794\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " Nov 02 20:14:25.165268 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.165255 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-xks-ca\") pod \"e332bca4-b357-4d7d-ba52-6563ce716794\" (UID: \"e332bca4-b357-4d7d-ba52-6563ce716794\") " Nov 02 20:14:25.165515 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.165487 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-openshift-ca" pod "e332bca4-b357-4d7d-ba52-6563ce716794" (UID: "e332bca4-b357-4d7d-ba52-6563ce716794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:25.165661 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.165642 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:25.167242 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.167218 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e332bca4-b357-4d7d-ba52-6563ce716794-kube-api-access-h7tbx" pod "e332bca4-b357-4d7d-ba52-6563ce716794" (UID: "e332bca4-b357-4d7d-ba52-6563ce716794"). InnerVolumeSpecName "kube-api-access-h7tbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:25.167362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.167328 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-xks-ca" pod "e332bca4-b357-4d7d-ba52-6563ce716794" (UID: "e332bca4-b357-4d7d-ba52-6563ce716794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:25.267275 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.267233 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e332bca4-b357-4d7d-ba52-6563ce716794-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:25.267434 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.267279 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7tbx\" (UniqueName: \"kubernetes.io/projected/e332bca4-b357-4d7d-ba52-6563ce716794-kube-api-access-h7tbx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:25.980243 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.980208 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" Nov 02 20:14:25.980243 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.980237 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8" event={"ID":"e332bca4-b357-4d7d-ba52-6563ce716794","Type":"ContainerDied","Data":"346daf931af419024efbca43f5d780dbf2a0d4308a5db067f68ed8f45c26783b"} Nov 02 20:14:25.980709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.980321 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346daf931af419024efbca43f5d780dbf2a0d4308a5db067f68ed8f45c26783b" Nov 02 20:14:25.982543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.982517 4966 generic.go:356] "Generic (PLEG): container finished" podID="ce845758-965b-496e-94a1-35908a53459d" containerID="0b82f4c553c987f2b052d4f4fcc4260bd655f9c708f6cc952990c729d4acf5b9" exitCode=0 Nov 02 20:14:25.982685 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:25.982549 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" event={"ID":"ce845758-965b-496e-94a1-35908a53459d","Type":"ContainerDied","Data":"0b82f4c553c987f2b052d4f4fcc4260bd655f9c708f6cc952990c729d4acf5b9"} Nov 02 20:14:27.132366 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.132340 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:27.187171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.187139 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce845758-965b-496e-94a1-35908a53459d-kube-api-access-zgbdq\" (UniqueName: \"kubernetes.io/projected/ce845758-965b-496e-94a1-35908a53459d-kube-api-access-zgbdq\") pod \"ce845758-965b-496e-94a1-35908a53459d\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " Nov 02 20:14:27.187397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.187196 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce845758-965b-496e-94a1-35908a53459d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce845758-965b-496e-94a1-35908a53459d-maas-api-openshift-ca\") pod \"ce845758-965b-496e-94a1-35908a53459d\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " Nov 02 20:14:27.187397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.187322 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce845758-965b-496e-94a1-35908a53459d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce845758-965b-496e-94a1-35908a53459d-maas-api-xks-ca\") pod \"ce845758-965b-496e-94a1-35908a53459d\" (UID: \"ce845758-965b-496e-94a1-35908a53459d\") " Nov 02 20:14:27.187674 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.187649 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce845758-965b-496e-94a1-35908a53459d-maas-api-openshift-ca" pod "ce845758-965b-496e-94a1-35908a53459d" (UID: "ce845758-965b-496e-94a1-35908a53459d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:27.189487 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.189452 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce845758-965b-496e-94a1-35908a53459d-maas-api-xks-ca" pod "ce845758-965b-496e-94a1-35908a53459d" (UID: "ce845758-965b-496e-94a1-35908a53459d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:27.189600 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.189505 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce845758-965b-496e-94a1-35908a53459d-kube-api-access-zgbdq" pod "ce845758-965b-496e-94a1-35908a53459d" (UID: "ce845758-965b-496e-94a1-35908a53459d"). InnerVolumeSpecName "kube-api-access-zgbdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:27.288912 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.288828 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbdq\" (UniqueName: \"kubernetes.io/projected/ce845758-965b-496e-94a1-35908a53459d-kube-api-access-zgbdq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:27.288912 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.288860 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce845758-965b-496e-94a1-35908a53459d-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:27.288912 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.288870 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce845758-965b-496e-94a1-35908a53459d-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:27.998170 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.998131 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" Nov 02 20:14:27.998373 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.998133 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h" event={"ID":"ce845758-965b-496e-94a1-35908a53459d","Type":"ContainerDied","Data":"a0f21bde8fa5fe015f03f92b05cf4c4e82a0175f4ae5fe2f8bcb9fd5b541d204"} Nov 02 20:14:27.998373 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:27.998266 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0f21bde8fa5fe015f03f92b05cf4c4e82a0175f4ae5fe2f8bcb9fd5b541d204" Nov 02 20:14:33.516820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:33.516778 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs"] Nov 02 20:14:35.889487 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.889443 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw"] Nov 02 20:14:35.889994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.889980 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ce845758-965b-496e-94a1-35908a53459d" containerName="revoke-keys" Nov 02 20:14:35.890046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.889996 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce845758-965b-496e-94a1-35908a53459d" containerName="revoke-keys" Nov 02 20:14:35.890046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.890023 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e332bca4-b357-4d7d-ba52-6563ce716794" containerName="revoke-keys" Nov 02 20:14:35.890046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.890028 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e332bca4-b357-4d7d-ba52-6563ce716794" containerName="revoke-keys" Nov 02 20:14:35.890156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.890103 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce845758-965b-496e-94a1-35908a53459d" containerName="revoke-keys" Nov 02 20:14:35.890156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.890115 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e332bca4-b357-4d7d-ba52-6563ce716794" containerName="revoke-keys" Nov 02 20:14:35.893764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.893746 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:35.944024 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.943998 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:14:35.989735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.989698 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:35.989735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.989704 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:14:35.990024 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.989936 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:14:35.991066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.991038 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:35.991222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.991089 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lfn\" (UniqueName: \"kubernetes.io/projected/16e2ed6b-8546-43a0-9207-e8f97d24aa84-kube-api-access-w8lfn\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:35.991408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.991388 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:35.994183 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:35.994150 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw"] Nov 02 20:14:36.092777 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.092742 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lfn\" (UniqueName: \"kubernetes.io/projected/16e2ed6b-8546-43a0-9207-e8f97d24aa84-kube-api-access-w8lfn\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:36.092970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.092878 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:36.092970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.092924 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:36.093696 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.093673 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:36.095425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.095393 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:36.145355 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.145262 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lfn\" (UniqueName: \"kubernetes.io/projected/16e2ed6b-8546-43a0-9207-e8f97d24aa84-kube-api-access-w8lfn\") pod \"maas-api-revoke-sub-d2dc8d2e-qxwvw\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:36.207865 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.207837 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:36.326620 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.326584 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" podUID="43ae073c-a362-4c94-84db-8b6ad37e1183" containerName="main" containerID="cri-o://65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e" gracePeriod=45 Nov 02 20:14:36.577833 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.577804 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw"] Nov 02 20:14:36.578547 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:14:36.578518 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e2ed6b_8546_43a0_9207_e8f97d24aa84.slice/crio-ebe462423151dba4b769882aaf5e92544b33652f4caf048c9217a3d56dcdb9ad WatchSource:0}: Error finding container ebe462423151dba4b769882aaf5e92544b33652f4caf048c9217a3d56dcdb9ad: Status 404 returned error can't find the container with id ebe462423151dba4b769882aaf5e92544b33652f4caf048c9217a3d56dcdb9ad Nov 02 20:14:36.585725 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.585698 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:14:36.701179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701093 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-tmp-dir\") pod \"43ae073c-a362-4c94-84db-8b6ad37e1183\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " Nov 02 20:14:36.701179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701143 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ae073c-a362-4c94-84db-8b6ad37e1183-kube-api-access-kgxlr\" (UniqueName: \"kubernetes.io/projected/43ae073c-a362-4c94-84db-8b6ad37e1183-kube-api-access-kgxlr\") pod \"43ae073c-a362-4c94-84db-8b6ad37e1183\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " Nov 02 20:14:36.701497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701252 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ae073c-a362-4c94-84db-8b6ad37e1183-tls-certs\" (UniqueName: \"kubernetes.io/secret/43ae073c-a362-4c94-84db-8b6ad37e1183-tls-certs\") pod \"43ae073c-a362-4c94-84db-8b6ad37e1183\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " Nov 02 20:14:36.701497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701277 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-dshm\") pod \"43ae073c-a362-4c94-84db-8b6ad37e1183\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " Nov 02 20:14:36.701497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701336 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-home\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-home\") pod \"43ae073c-a362-4c94-84db-8b6ad37e1183\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " Nov 02 20:14:36.701497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701370 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-model-cache\") pod \"43ae073c-a362-4c94-84db-8b6ad37e1183\" (UID: \"43ae073c-a362-4c94-84db-8b6ad37e1183\") " Nov 02 20:14:36.701497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701410 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-tmp-dir" pod "43ae073c-a362-4c94-84db-8b6ad37e1183" (UID: "43ae073c-a362-4c94-84db-8b6ad37e1183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:36.702430 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701840 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-home" pod "43ae073c-a362-4c94-84db-8b6ad37e1183" (UID: "43ae073c-a362-4c94-84db-8b6ad37e1183"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:36.702430 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701862 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:36.702430 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.701931 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-model-cache" pod "43ae073c-a362-4c94-84db-8b6ad37e1183" (UID: "43ae073c-a362-4c94-84db-8b6ad37e1183"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:36.703739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.703717 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ae073c-a362-4c94-84db-8b6ad37e1183-tls-certs" pod "43ae073c-a362-4c94-84db-8b6ad37e1183" (UID: "43ae073c-a362-4c94-84db-8b6ad37e1183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:36.703983 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.703962 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-dshm" pod "43ae073c-a362-4c94-84db-8b6ad37e1183" (UID: "43ae073c-a362-4c94-84db-8b6ad37e1183"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:14:36.704073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.704045 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ae073c-a362-4c94-84db-8b6ad37e1183-kube-api-access-kgxlr" pod "43ae073c-a362-4c94-84db-8b6ad37e1183" (UID: "43ae073c-a362-4c94-84db-8b6ad37e1183"). InnerVolumeSpecName "kube-api-access-kgxlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:36.802827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.802789 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ae073c-a362-4c94-84db-8b6ad37e1183-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:36.802827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.802824 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:36.802827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.802833 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:36.803166 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.802842 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43ae073c-a362-4c94-84db-8b6ad37e1183-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:36.803166 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:36.802851 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgxlr\" (UniqueName: \"kubernetes.io/projected/43ae073c-a362-4c94-84db-8b6ad37e1183-kube-api-access-kgxlr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:37.062896 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.062860 4966 generic.go:356] "Generic (PLEG): container finished" podID="43ae073c-a362-4c94-84db-8b6ad37e1183" containerID="65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e" exitCode=0 Nov 02 20:14:37.063328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.062908 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" Nov 02 20:14:37.063328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.062931 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" event={"ID":"43ae073c-a362-4c94-84db-8b6ad37e1183","Type":"ContainerDied","Data":"65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e"} Nov 02 20:14:37.063328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.062981 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw" event={"ID":"43ae073c-a362-4c94-84db-8b6ad37e1183","Type":"ContainerDied","Data":"6674e27c70a2bcea0abae8b66e859a1537ba057c20655c6c0df4657ff5ffe525"} Nov 02 20:14:37.063328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.063002 4966 scope.go:122] "RemoveContainer" containerID="65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e" Nov 02 20:14:37.064976 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.064949 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" event={"ID":"16e2ed6b-8546-43a0-9207-e8f97d24aa84","Type":"ContainerStarted","Data":"ffa41b58073908eaa64dbf6e36e74000eabc882e6e0454b42de1f2e5cdd5984d"} Nov 02 20:14:37.065100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.064999 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" event={"ID":"16e2ed6b-8546-43a0-9207-e8f97d24aa84","Type":"ContainerStarted","Data":"ebe462423151dba4b769882aaf5e92544b33652f4caf048c9217a3d56dcdb9ad"} Nov 02 20:14:37.075108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.075088 4966 scope.go:122] "RemoveContainer" containerID="65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e" Nov 02 20:14:37.075380 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:14:37.075360 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e\": container with ID starting with 65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e not found: ID does not exist" containerID="65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e" Nov 02 20:14:37.075453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.075392 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e"} err="failed to get container status \"65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e\": rpc error: code = NotFound desc = could not find container \"65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e\": container with ID starting with 65638a258bafcb6a00cd9dc5661d013d686853da5df900e6fa07cf3125f0717e not found: ID does not exist" Nov 02 20:14:37.220509 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.220478 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw"] Nov 02 20:14:37.223653 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.223626 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-28fa24/test-model-28fa24-kserve-7d4d54484f-vnkfw"] Nov 02 20:14:37.324936 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:37.324828 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" podStartSLOduration=2.324812841 podStartE2EDuration="2.324812841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:37.322751079 +0000 UTC m=+2783.271406917" watchObservedRunningTime="2026-11-02 20:14:37.324812841 +0000 UTC m=+2783.273468680" Nov 02 20:14:38.074900 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:38.074863 4966 generic.go:356] "Generic (PLEG): container finished" podID="16e2ed6b-8546-43a0-9207-e8f97d24aa84" containerID="ffa41b58073908eaa64dbf6e36e74000eabc882e6e0454b42de1f2e5cdd5984d" exitCode=0 Nov 02 20:14:38.075379 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:38.074947 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" event={"ID":"16e2ed6b-8546-43a0-9207-e8f97d24aa84","Type":"ContainerDied","Data":"ffa41b58073908eaa64dbf6e36e74000eabc882e6e0454b42de1f2e5cdd5984d"} Nov 02 20:14:38.186117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:38.186078 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ae073c-a362-4c94-84db-8b6ad37e1183" path="/var/lib/kubelet/pods/43ae073c-a362-4c94-84db-8b6ad37e1183/volumes" Nov 02 20:14:38.517330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:38.517248 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" podUID="e62b3cb3-a13a-4404-9c9a-f33a21110843" containerName="payload-pre-processing" containerID="cri-o://515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f" gracePeriod=25 Nov 02 20:14:39.233937 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.233900 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:39.328035 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.327984 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e2ed6b-8546-43a0-9207-e8f97d24aa84-kube-api-access-w8lfn\" (UniqueName: \"kubernetes.io/projected/16e2ed6b-8546-43a0-9207-e8f97d24aa84-kube-api-access-w8lfn\") pod \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " Nov 02 20:14:39.328258 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.328092 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-xks-ca\") pod \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " Nov 02 20:14:39.328258 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.328151 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-openshift-ca\") pod \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\" (UID: \"16e2ed6b-8546-43a0-9207-e8f97d24aa84\") " Nov 02 20:14:39.328615 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.328583 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-openshift-ca" pod "16e2ed6b-8546-43a0-9207-e8f97d24aa84" (UID: "16e2ed6b-8546-43a0-9207-e8f97d24aa84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:39.328753 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.328706 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:39.330420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.330392 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-xks-ca" pod "16e2ed6b-8546-43a0-9207-e8f97d24aa84" (UID: "16e2ed6b-8546-43a0-9207-e8f97d24aa84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:39.330420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.330412 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e2ed6b-8546-43a0-9207-e8f97d24aa84-kube-api-access-w8lfn" pod "16e2ed6b-8546-43a0-9207-e8f97d24aa84" (UID: "16e2ed6b-8546-43a0-9207-e8f97d24aa84"). InnerVolumeSpecName "kube-api-access-w8lfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:39.430353 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.430317 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8lfn\" (UniqueName: \"kubernetes.io/projected/16e2ed6b-8546-43a0-9207-e8f97d24aa84-kube-api-access-w8lfn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:39.430353 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:39.430348 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e2ed6b-8546-43a0-9207-e8f97d24aa84-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:40.092434 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:40.092405 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" Nov 02 20:14:40.092630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:40.092448 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw" event={"ID":"16e2ed6b-8546-43a0-9207-e8f97d24aa84","Type":"ContainerDied","Data":"ebe462423151dba4b769882aaf5e92544b33652f4caf048c9217a3d56dcdb9ad"} Nov 02 20:14:40.092630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:40.092542 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe462423151dba4b769882aaf5e92544b33652f4caf048c9217a3d56dcdb9ad" Nov 02 20:14:43.721283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:43.721240 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" podUID="e62b3cb3-a13a-4404-9c9a-f33a21110843" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:14:44.753527 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:14:44.753486 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:14:53.721697 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:53.721656 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" podUID="e62b3cb3-a13a-4404-9c9a-f33a21110843" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:14:57.280628 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.280589 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6"] Nov 02 20:14:57.752039 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.751999 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5"] Nov 02 20:14:57.752755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.752738 4966 cpu_manager.go:414] "containerMap: removing container" podUID="16e2ed6b-8546-43a0-9207-e8f97d24aa84" containerName="revoke-keys" Nov 02 20:14:57.752801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.752758 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e2ed6b-8546-43a0-9207-e8f97d24aa84" containerName="revoke-keys" Nov 02 20:14:57.752849 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.752815 4966 cpu_manager.go:414] "containerMap: removing container" podUID="43ae073c-a362-4c94-84db-8b6ad37e1183" containerName="main" Nov 02 20:14:57.752849 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.752823 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ae073c-a362-4c94-84db-8b6ad37e1183" containerName="main" Nov 02 20:14:57.752967 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.752955 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ae073c-a362-4c94-84db-8b6ad37e1183" containerName="main" Nov 02 20:14:57.753005 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.752973 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e2ed6b-8546-43a0-9207-e8f97d24aa84" containerName="revoke-keys" Nov 02 20:14:57.757020 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.757001 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.805855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.805812 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:14:57.809023 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.808992 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5"] Nov 02 20:14:57.821345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.821270 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:57.821345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.821311 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:14:57.842932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.842901 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6tgx\" (UniqueName: \"kubernetes.io/projected/28cab378-908f-4877-af24-685c7d93e7ae-kube-api-access-t6tgx\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.843058 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.842967 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.843144 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.843116 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.850481 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.850460 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:14:57.944100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.944062 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.944331 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.944142 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.944331 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.944280 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tgx\" (UniqueName: \"kubernetes.io/projected/28cab378-908f-4877-af24-685c7d93e7ae-kube-api-access-t6tgx\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.944949 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.944912 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.946713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.946682 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:57.997589 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:57.997564 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6tgx\" (UniqueName: \"kubernetes.io/projected/28cab378-908f-4877-af24-685c7d93e7ae-kube-api-access-t6tgx\") pod \"maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:58.071711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.071612 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:14:58.449957 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.449914 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5"] Nov 02 20:14:58.451661 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:14:58.451636 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cab378_908f_4877_af24_685c7d93e7ae.slice/crio-3141221632ca3b7508b0df13bcb9bd108420610e39121d6b8d87dda21605399c WatchSource:0}: Error finding container 3141221632ca3b7508b0df13bcb9bd108420610e39121d6b8d87dda21605399c: Status 404 returned error can't find the container with id 3141221632ca3b7508b0df13bcb9bd108420610e39121d6b8d87dda21605399c Nov 02 20:14:58.778886 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.778859 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:14:58.857129 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.857103 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62b3cb3-a13a-4404-9c9a-f33a21110843-kube-api-access-m86cp\" (UniqueName: \"kubernetes.io/projected/e62b3cb3-a13a-4404-9c9a-f33a21110843-kube-api-access-m86cp\") pod \"e62b3cb3-a13a-4404-9c9a-f33a21110843\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " Nov 02 20:14:58.857253 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.857239 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62b3cb3-a13a-4404-9c9a-f33a21110843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62b3cb3-a13a-4404-9c9a-f33a21110843-plugins-config-volume\") pod \"e62b3cb3-a13a-4404-9c9a-f33a21110843\" (UID: \"e62b3cb3-a13a-4404-9c9a-f33a21110843\") " Nov 02 20:14:58.857601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.857578 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62b3cb3-a13a-4404-9c9a-f33a21110843-plugins-config-volume" pod "e62b3cb3-a13a-4404-9c9a-f33a21110843" (UID: "e62b3cb3-a13a-4404-9c9a-f33a21110843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:58.859261 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.859239 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62b3cb3-a13a-4404-9c9a-f33a21110843-kube-api-access-m86cp" pod "e62b3cb3-a13a-4404-9c9a-f33a21110843" (UID: "e62b3cb3-a13a-4404-9c9a-f33a21110843"). InnerVolumeSpecName "kube-api-access-m86cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:58.959359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.959325 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62b3cb3-a13a-4404-9c9a-f33a21110843-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:58.959478 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:58.959364 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86cp\" (UniqueName: \"kubernetes.io/projected/e62b3cb3-a13a-4404-9c9a-f33a21110843-kube-api-access-m86cp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:59.233087 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.233051 4966 generic.go:356] "Generic (PLEG): container finished" podID="e62b3cb3-a13a-4404-9c9a-f33a21110843" containerID="515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f" exitCode=0 Nov 02 20:14:59.233352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.233102 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" Nov 02 20:14:59.233352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.233136 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" event={"ID":"e62b3cb3-a13a-4404-9c9a-f33a21110843","Type":"ContainerDied","Data":"515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f"} Nov 02 20:14:59.233352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.233199 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs" event={"ID":"e62b3cb3-a13a-4404-9c9a-f33a21110843","Type":"ContainerDied","Data":"de4c00a261b7d27039316288ec7cc5e11198f9693893ad21033eb250f3fb9711"} Nov 02 20:14:59.233352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.233229 4966 scope.go:122] "RemoveContainer" containerID="515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f" Nov 02 20:14:59.235639 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.235613 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" event={"ID":"28cab378-908f-4877-af24-685c7d93e7ae","Type":"ContainerStarted","Data":"0edbdac4538de16819e51829a566c198e0fc3389ca2631806ccd7ccf2725c55b"} Nov 02 20:14:59.235852 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.235826 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" event={"ID":"28cab378-908f-4877-af24-685c7d93e7ae","Type":"ContainerStarted","Data":"3141221632ca3b7508b0df13bcb9bd108420610e39121d6b8d87dda21605399c"} Nov 02 20:14:59.245750 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.245725 4966 scope.go:122] "RemoveContainer" containerID="515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f" Nov 02 20:14:59.246072 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:14:59.246051 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f\": container with ID starting with 515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f not found: ID does not exist" containerID="515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f" Nov 02 20:14:59.246146 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.246086 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f"} err="failed to get container status \"515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f\": rpc error: code = NotFound desc = could not find container \"515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f\": container with ID starting with 515d0710a231e03c1f609ffd56738c892421da530d82ea20014c0f66d476b30f not found: ID does not exist" Nov 02 20:14:59.387091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.387057 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs"] Nov 02 20:14:59.433138 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.433108 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-838bc1-549bfd4d6c-zdczs"] Nov 02 20:14:59.534887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:14:59.534790 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" podStartSLOduration=2.534777127 podStartE2EDuration="2.534777127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:59.532943621 +0000 UTC m=+2805.481599459" watchObservedRunningTime="2026-11-02 20:14:59.534777127 +0000 UTC m=+2805.483432967" Nov 02 20:15:00.157223 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.157187 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr"] Nov 02 20:15:00.157718 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.157702 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e62b3cb3-a13a-4404-9c9a-f33a21110843" containerName="payload-pre-processing" Nov 02 20:15:00.157803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.157719 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62b3cb3-a13a-4404-9c9a-f33a21110843" containerName="payload-pre-processing" Nov 02 20:15:00.157886 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.157873 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62b3cb3-a13a-4404-9c9a-f33a21110843" containerName="payload-pre-processing" Nov 02 20:15:00.161417 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.161393 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.161627 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.161602 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n"] Nov 02 20:15:00.166178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.166156 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv"] Nov 02 20:15:00.166362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.166292 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.169955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.169931 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.185663 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.185635 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62b3cb3-a13a-4404-9c9a-f33a21110843" path="/var/lib/kubelet/pods/e62b3cb3-a13a-4404-9c9a-f33a21110843/volumes" Nov 02 20:15:00.204458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.204420 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl"] Nov 02 20:15:00.208851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.208821 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg"] Nov 02 20:15:00.209003 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.208982 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.212894 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.212867 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w"] Nov 02 20:15:00.213020 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.212975 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.216699 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.216678 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.216813 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.216706 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh"] Nov 02 20:15:00.220503 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.220484 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.220711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.220699 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm"] Nov 02 20:15:00.224430 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.224407 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc"] Nov 02 20:15:00.224589 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.224571 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.230070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.230043 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr"] Nov 02 20:15:00.230190 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.230171 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.234156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.234134 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.247458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.247423 4966 generic.go:356] "Generic (PLEG): container finished" podID="28cab378-908f-4877-af24-685c7d93e7ae" containerID="0edbdac4538de16819e51829a566c198e0fc3389ca2631806ccd7ccf2725c55b" exitCode=0 Nov 02 20:15:00.247739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.247494 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" event={"ID":"28cab378-908f-4877-af24-685c7d93e7ae","Type":"ContainerDied","Data":"0edbdac4538de16819e51829a566c198e0fc3389ca2631806ccd7ccf2725c55b"} Nov 02 20:15:00.251345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.251319 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9"] Nov 02 20:15:00.256047 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.255877 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:00.256896 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.256869 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr"] Nov 02 20:15:00.260389 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.260364 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv"] Nov 02 20:15:00.274792 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.274752 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.274927 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.274812 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df678d54-f0bd-4b95-b86a-2c67ae313fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.274927 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.274895 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvztz\" (UniqueName: \"kubernetes.io/projected/d04eecc2-cc32-4536-8a4b-de0b553306c1-kube-api-access-kvztz\") pod \"maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.275011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.274960 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffk6\" (UniqueName: \"kubernetes.io/projected/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-kube-api-access-zffk6\") pod \"maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.275011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.274994 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04eecc2-cc32-4536-8a4b-de0b553306c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.275092 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275027 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.275202 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275178 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.275251 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275213 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsdw\" (UniqueName: \"kubernetes.io/projected/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-kube-api-access-dwsdw\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.275251 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275238 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331eb841-ecb1-4fca-813e-842e3e9f0529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.275388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275349 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.275435 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275391 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7t5j\" (UniqueName: \"kubernetes.io/projected/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-kube-api-access-q7t5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.275435 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275411 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjd27\" (UniqueName: \"kubernetes.io/projected/df678d54-f0bd-4b95-b86a-2c67ae313fcd-kube-api-access-sjd27\") pod \"maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.275652 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275629 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztn6\" (UniqueName: \"kubernetes.io/projected/5addd05e-f0bb-40aa-a86c-dce2c22129fb-kube-api-access-jztn6\") pod \"maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.275743 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275693 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqmb\" (UniqueName: \"kubernetes.io/projected/331eb841-ecb1-4fca-813e-842e3e9f0529-kube-api-access-szqmb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.275743 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275729 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dzz\" (UniqueName: \"kubernetes.io/projected/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-kube-api-access-r5dzz\") pod \"maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.275854 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.275757 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5addd05e-f0bb-40aa-a86c-dce2c22129fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.302367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.300669 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg"] Nov 02 20:15:00.307321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.306787 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n"] Nov 02 20:15:00.311781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.311648 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl"] Nov 02 20:15:00.320647 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.320614 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc"] Nov 02 20:15:00.349350 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.349285 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr"] Nov 02 20:15:00.354263 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.354230 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9"] Nov 02 20:15:00.357503 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.357444 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm"] Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.377464 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztn6\" (UniqueName: \"kubernetes.io/projected/5addd05e-f0bb-40aa-a86c-dce2c22129fb-kube-api-access-jztn6\") pod \"maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.377539 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqmb\" (UniqueName: \"kubernetes.io/projected/331eb841-ecb1-4fca-813e-842e3e9f0529-kube-api-access-szqmb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.377578 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dzz\" (UniqueName: \"kubernetes.io/projected/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-kube-api-access-r5dzz\") pod \"maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.377611 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5addd05e-f0bb-40aa-a86c-dce2c22129fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.377657 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.378442 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5addd05e-f0bb-40aa-a86c-dce2c22129fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.379147 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.379201 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df678d54-f0bd-4b95-b86a-2c67ae313fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.379233 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvztz\" (UniqueName: \"kubernetes.io/projected/d04eecc2-cc32-4536-8a4b-de0b553306c1-kube-api-access-kvztz\") pod \"maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.379266 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffk6\" (UniqueName: \"kubernetes.io/projected/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-kube-api-access-zffk6\") pod \"maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.379317 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04eecc2-cc32-4536-8a4b-de0b553306c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386029 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386090 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249dw\" (UniqueName: \"kubernetes.io/projected/d3a3287b-f705-4b73-aed1-5c04e4febbc2-kube-api-access-249dw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386140 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdpf\" (UniqueName: \"kubernetes.io/projected/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-kube-api-access-9xdpf\") pod \"maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386197 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386231 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsdw\" (UniqueName: \"kubernetes.io/projected/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-kube-api-access-dwsdw\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386267 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksk8\" (UniqueName: \"kubernetes.io/projected/a825edf8-6b57-40d8-81a4-873300069d62-kube-api-access-6ksk8\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386323 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331eb841-ecb1-4fca-813e-842e3e9f0529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386386 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386432 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a825edf8-6b57-40d8-81a4-873300069d62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386477 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7t5j\" (UniqueName: \"kubernetes.io/projected/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-kube-api-access-q7t5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386511 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjd27\" (UniqueName: \"kubernetes.io/projected/df678d54-f0bd-4b95-b86a-2c67ae313fcd-kube-api-access-sjd27\") pod \"maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.386586 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a3287b-f705-4b73-aed1-5c04e4febbc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.387432 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.388359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.388051 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df678d54-f0bd-4b95-b86a-2c67ae313fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.394100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.389182 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w"] Nov 02 20:15:00.394100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.390015 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.394100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.390174 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331eb841-ecb1-4fca-813e-842e3e9f0529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.394100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.390739 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.394100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.390833 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.394100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.393336 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04eecc2-cc32-4536-8a4b-de0b553306c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.402365 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.397354 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh"] Nov 02 20:15:00.431505 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.431425 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqmb\" (UniqueName: \"kubernetes.io/projected/331eb841-ecb1-4fca-813e-842e3e9f0529-kube-api-access-szqmb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.431660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.431528 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dzz\" (UniqueName: \"kubernetes.io/projected/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-kube-api-access-r5dzz\") pod \"maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.431660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.431528 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztn6\" (UniqueName: \"kubernetes.io/projected/5addd05e-f0bb-40aa-a86c-dce2c22129fb-kube-api-access-jztn6\") pod \"maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.445755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.445690 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvztz\" (UniqueName: \"kubernetes.io/projected/d04eecc2-cc32-4536-8a4b-de0b553306c1-kube-api-access-kvztz\") pod \"maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.446540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.446506 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjd27\" (UniqueName: \"kubernetes.io/projected/df678d54-f0bd-4b95-b86a-2c67ae313fcd-kube-api-access-sjd27\") pod \"maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.446540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.446530 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7t5j\" (UniqueName: \"kubernetes.io/projected/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-kube-api-access-q7t5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.446702 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.446578 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsdw\" (UniqueName: \"kubernetes.io/projected/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-kube-api-access-dwsdw\") pod \"maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.447208 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.447184 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffk6\" (UniqueName: \"kubernetes.io/projected/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-kube-api-access-zffk6\") pod \"maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.477420 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.477389 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:00.486923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.486891 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:00.487363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.487338 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249dw\" (UniqueName: \"kubernetes.io/projected/d3a3287b-f705-4b73-aed1-5c04e4febbc2-kube-api-access-249dw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.487475 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.487397 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdpf\" (UniqueName: \"kubernetes.io/projected/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-kube-api-access-9xdpf\") pod \"maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:00.487475 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.487458 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksk8\" (UniqueName: \"kubernetes.io/projected/a825edf8-6b57-40d8-81a4-873300069d62-kube-api-access-6ksk8\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.487602 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.487536 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a825edf8-6b57-40d8-81a4-873300069d62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.487658 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.487622 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a3287b-f705-4b73-aed1-5c04e4febbc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.487749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.487734 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:00.488601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.488572 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:00.488805 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.488778 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a3287b-f705-4b73-aed1-5c04e4febbc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.488805 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.488800 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a825edf8-6b57-40d8-81a4-873300069d62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.494773 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.494751 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:00.527735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.527685 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:00.541061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.541028 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-249dw\" (UniqueName: \"kubernetes.io/projected/d3a3287b-f705-4b73-aed1-5c04e4febbc2-kube-api-access-249dw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.541773 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.541630 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdpf\" (UniqueName: \"kubernetes.io/projected/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-kube-api-access-9xdpf\") pod \"maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:00.542654 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.542621 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksk8\" (UniqueName: \"kubernetes.io/projected/a825edf8-6b57-40d8-81a4-873300069d62-kube-api-access-6ksk8\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.544444 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.544414 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:00.553016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.552274 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:00.564549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.564348 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:00.569828 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.569636 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:00.579501 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.579468 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:00.587867 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.586236 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:00.592810 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:00.592721 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:01.067347 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.067315 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr"] Nov 02 20:15:01.068047 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.068020 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a68f258_f099_4434_bac3_e3f9f7a3bfdf.slice/crio-159209fdec2d00de3c84367c9bfb7b30f3a08d0880667b454b18e0dcebd4eef3 WatchSource:0}: Error finding container 159209fdec2d00de3c84367c9bfb7b30f3a08d0880667b454b18e0dcebd4eef3: Status 404 returned error can't find the container with id 159209fdec2d00de3c84367c9bfb7b30f3a08d0880667b454b18e0dcebd4eef3 Nov 02 20:15:01.114417 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.113423 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n"] Nov 02 20:15:01.114417 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.113656 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04eecc2_cc32_4536_8a4b_de0b553306c1.slice/crio-98827974d10a501546b0cd91450647bf9c8b5661fb6886a1c463d9926f37d7b5 WatchSource:0}: Error finding container 98827974d10a501546b0cd91450647bf9c8b5661fb6886a1c463d9926f37d7b5: Status 404 returned error can't find the container with id 98827974d10a501546b0cd91450647bf9c8b5661fb6886a1c463d9926f37d7b5 Nov 02 20:15:01.116595 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.116572 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfa89f3_ad5d_42d4_9ab8_5f3f11c167a0.slice/crio-36ecf0105b9dae3229ddba13d3c36985132b363a42714838468a7323fe34e422 WatchSource:0}: Error finding container 36ecf0105b9dae3229ddba13d3c36985132b363a42714838468a7323fe34e422: Status 404 returned error can't find the container with id 36ecf0105b9dae3229ddba13d3c36985132b363a42714838468a7323fe34e422 Nov 02 20:15:01.118691 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.118669 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv"] Nov 02 20:15:01.160907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.160872 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm"] Nov 02 20:15:01.162293 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.162269 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74b7ad1_f775_4370_bb1a_4d71643eaf6f.slice/crio-982e70940d4e44978a86b922f9f0f0623c111ec6004ea59a5711d7d1beaac483 WatchSource:0}: Error finding container 982e70940d4e44978a86b922f9f0f0623c111ec6004ea59a5711d7d1beaac483: Status 404 returned error can't find the container with id 982e70940d4e44978a86b922f9f0f0623c111ec6004ea59a5711d7d1beaac483 Nov 02 20:15:01.170003 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.169967 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf678d54_f0bd_4b95_b86a_2c67ae313fcd.slice/crio-d3d2908a44aa0495b35dad5010fd5503975c7be79e7764ada42434583370406d WatchSource:0}: Error finding container d3d2908a44aa0495b35dad5010fd5503975c7be79e7764ada42434583370406d: Status 404 returned error can't find the container with id d3d2908a44aa0495b35dad5010fd5503975c7be79e7764ada42434583370406d Nov 02 20:15:01.170750 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.170726 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg"] Nov 02 20:15:01.186282 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.186258 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl"] Nov 02 20:15:01.186832 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.186809 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331eb841_ecb1_4fca_813e_842e3e9f0529.slice/crio-9d11ce58fe2980c9e99193fc83179298ccdc96fbdec5121dfcbadc185f67a2c4 WatchSource:0}: Error finding container 9d11ce58fe2980c9e99193fc83179298ccdc96fbdec5121dfcbadc185f67a2c4: Status 404 returned error can't find the container with id 9d11ce58fe2980c9e99193fc83179298ccdc96fbdec5121dfcbadc185f67a2c4 Nov 02 20:15:01.204503 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.204472 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w"] Nov 02 20:15:01.206141 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.206113 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5addd05e_f0bb_40aa_a86c_dce2c22129fb.slice/crio-fed2053a6bec52e1657858646640c9c9237cb9b99191e3e404587a49e26eb2ce WatchSource:0}: Error finding container fed2053a6bec52e1657858646640c9c9237cb9b99191e3e404587a49e26eb2ce: Status 404 returned error can't find the container with id fed2053a6bec52e1657858646640c9c9237cb9b99191e3e404587a49e26eb2ce Nov 02 20:15:01.259470 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.259442 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9"] Nov 02 20:15:01.266485 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.266452 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" event={"ID":"5addd05e-f0bb-40aa-a86c-dce2c22129fb","Type":"ContainerStarted","Data":"fed2053a6bec52e1657858646640c9c9237cb9b99191e3e404587a49e26eb2ce"} Nov 02 20:15:01.267812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.267792 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerStarted","Data":"36ecf0105b9dae3229ddba13d3c36985132b363a42714838468a7323fe34e422"} Nov 02 20:15:01.269428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.269405 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" event={"ID":"5a68f258-f099-4434-bac3-e3f9f7a3bfdf","Type":"ContainerStarted","Data":"159209fdec2d00de3c84367c9bfb7b30f3a08d0880667b454b18e0dcebd4eef3"} Nov 02 20:15:01.270881 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.270833 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" event={"ID":"331eb841-ecb1-4fca-813e-842e3e9f0529","Type":"ContainerStarted","Data":"9d11ce58fe2980c9e99193fc83179298ccdc96fbdec5121dfcbadc185f67a2c4"} Nov 02 20:15:01.271976 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.271954 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerStarted","Data":"d3d2908a44aa0495b35dad5010fd5503975c7be79e7764ada42434583370406d"} Nov 02 20:15:01.273207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.273186 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" event={"ID":"f74b7ad1-f775-4370-bb1a-4d71643eaf6f","Type":"ContainerStarted","Data":"982e70940d4e44978a86b922f9f0f0623c111ec6004ea59a5711d7d1beaac483"} Nov 02 20:15:01.274912 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.274887 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" event={"ID":"d04eecc2-cc32-4536-8a4b-de0b553306c1","Type":"ContainerStarted","Data":"98827974d10a501546b0cd91450647bf9c8b5661fb6886a1c463d9926f37d7b5"} Nov 02 20:15:01.278095 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.278075 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr"] Nov 02 20:15:01.294851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.294828 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc"] Nov 02 20:15:01.298045 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.298026 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh"] Nov 02 20:15:01.344927 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.344395 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971c7ec4_2c35_44c3_afdb_6e2e9df1a7b1.slice/crio-6bd327167a85ceb5879021e7c19c946481ca9718037f8bf507e6311acdd7e89c WatchSource:0}: Error finding container 6bd327167a85ceb5879021e7c19c946481ca9718037f8bf507e6311acdd7e89c: Status 404 returned error can't find the container with id 6bd327167a85ceb5879021e7c19c946481ca9718037f8bf507e6311acdd7e89c Nov 02 20:15:01.444990 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.444947 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a3287b_f705_4b73_aed1_5c04e4febbc2.slice/crio-97d648d28a21190bd00587a8422eb6ad6bf2df89dee15c29189351ad24289eef WatchSource:0}: Error finding container 97d648d28a21190bd00587a8422eb6ad6bf2df89dee15c29189351ad24289eef: Status 404 returned error can't find the container with id 97d648d28a21190bd00587a8422eb6ad6bf2df89dee15c29189351ad24289eef Nov 02 20:15:01.446403 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:01.446371 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda825edf8_6b57_40d8_81a4_873300069d62.slice/crio-04405ebc980ebf0254f0d00c34c4437e0a7d0ac496a3694b79353a353764d39d WatchSource:0}: Error finding container 04405ebc980ebf0254f0d00c34c4437e0a7d0ac496a3694b79353a353764d39d: Status 404 returned error can't find the container with id 04405ebc980ebf0254f0d00c34c4437e0a7d0ac496a3694b79353a353764d39d Nov 02 20:15:01.714775 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.714746 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:15:01.805112 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.805071 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-openshift-ca\") pod \"28cab378-908f-4877-af24-685c7d93e7ae\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " Nov 02 20:15:01.805308 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.805122 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cab378-908f-4877-af24-685c7d93e7ae-kube-api-access-t6tgx\" (UniqueName: \"kubernetes.io/projected/28cab378-908f-4877-af24-685c7d93e7ae-kube-api-access-t6tgx\") pod \"28cab378-908f-4877-af24-685c7d93e7ae\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " Nov 02 20:15:01.805308 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.805227 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-xks-ca\") pod \"28cab378-908f-4877-af24-685c7d93e7ae\" (UID: \"28cab378-908f-4877-af24-685c7d93e7ae\") " Nov 02 20:15:01.805573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.805547 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-openshift-ca" pod "28cab378-908f-4877-af24-685c7d93e7ae" (UID: "28cab378-908f-4877-af24-685c7d93e7ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:01.848248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.848134 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-xks-ca" pod "28cab378-908f-4877-af24-685c7d93e7ae" (UID: "28cab378-908f-4877-af24-685c7d93e7ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:01.850397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.850352 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cab378-908f-4877-af24-685c7d93e7ae-kube-api-access-t6tgx" pod "28cab378-908f-4877-af24-685c7d93e7ae" (UID: "28cab378-908f-4877-af24-685c7d93e7ae"). InnerVolumeSpecName "kube-api-access-t6tgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:01.906644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.906613 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:01.906644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.906645 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28cab378-908f-4877-af24-685c7d93e7ae-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:01.906837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:01.906661 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6tgx\" (UniqueName: \"kubernetes.io/projected/28cab378-908f-4877-af24-685c7d93e7ae-kube-api-access-t6tgx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:02.284738 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.284701 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerStarted","Data":"f72567e987d0f25bc8c605867c6491e5f1e3af19a6556975b0141db94a8d1f27"} Nov 02 20:15:02.286792 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.286767 4966 generic.go:356] "Generic (PLEG): container finished" podID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerID="d516dd63c2a360e006df0580d304238273a89e62f0816586e6dcd67b6ddda497" exitCode=60 Nov 02 20:15:02.286918 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.286825 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" event={"ID":"f74b7ad1-f775-4370-bb1a-4d71643eaf6f","Type":"ContainerDied","Data":"d516dd63c2a360e006df0580d304238273a89e62f0816586e6dcd67b6ddda497"} Nov 02 20:15:02.287073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.287058 4966 scope.go:122] "RemoveContainer" containerID="d516dd63c2a360e006df0580d304238273a89e62f0816586e6dcd67b6ddda497" Nov 02 20:15:02.288614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.288583 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" event={"ID":"d3a3287b-f705-4b73-aed1-5c04e4febbc2","Type":"ContainerStarted","Data":"299fcb8d6a891b50b45507909cf4642d985cf2453d4f0f51ac902b961bcfbed3"} Nov 02 20:15:02.288715 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.288635 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" event={"ID":"d3a3287b-f705-4b73-aed1-5c04e4febbc2","Type":"ContainerStarted","Data":"97d648d28a21190bd00587a8422eb6ad6bf2df89dee15c29189351ad24289eef"} Nov 02 20:15:02.296586 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.293693 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" event={"ID":"28cab378-908f-4877-af24-685c7d93e7ae","Type":"ContainerDied","Data":"3141221632ca3b7508b0df13bcb9bd108420610e39121d6b8d87dda21605399c"} Nov 02 20:15:02.296586 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.293795 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3141221632ca3b7508b0df13bcb9bd108420610e39121d6b8d87dda21605399c" Nov 02 20:15:02.296586 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.293887 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5" Nov 02 20:15:02.307519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.307473 4966 generic.go:356] "Generic (PLEG): container finished" podID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerID="ad8eb08593a7710407b00f22c536976d7464461bb1be9749d2f9a472cd5b6158" exitCode=60 Nov 02 20:15:02.307676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.307566 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" event={"ID":"d04eecc2-cc32-4536-8a4b-de0b553306c1","Type":"ContainerDied","Data":"ad8eb08593a7710407b00f22c536976d7464461bb1be9749d2f9a472cd5b6158"} Nov 02 20:15:02.308054 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.308032 4966 scope.go:122] "RemoveContainer" containerID="ad8eb08593a7710407b00f22c536976d7464461bb1be9749d2f9a472cd5b6158" Nov 02 20:15:02.312895 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.312853 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" event={"ID":"a825edf8-6b57-40d8-81a4-873300069d62","Type":"ContainerStarted","Data":"e02e424a5ea631eef6366824eb366f4b7deacf3231e1b88fd146f6f54689b970"} Nov 02 20:15:02.313001 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.312902 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" event={"ID":"a825edf8-6b57-40d8-81a4-873300069d62","Type":"ContainerStarted","Data":"04405ebc980ebf0254f0d00c34c4437e0a7d0ac496a3694b79353a353764d39d"} Nov 02 20:15:02.315575 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.315551 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" event={"ID":"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1","Type":"ContainerStarted","Data":"1e674b05f00f6e87751e74ae48de898cf33bb9f45d8f0026f0c7867e77277845"} Nov 02 20:15:02.315728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.315707 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" event={"ID":"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1","Type":"ContainerStarted","Data":"6bd327167a85ceb5879021e7c19c946481ca9718037f8bf507e6311acdd7e89c"} Nov 02 20:15:02.317205 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.317181 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" event={"ID":"5addd05e-f0bb-40aa-a86c-dce2c22129fb","Type":"ContainerStarted","Data":"c9e4cfc21f6e00f783a83253cc4f670e96ea64b35a7b14d5d8a3022c2fe3553e"} Nov 02 20:15:02.318975 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.318948 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerStarted","Data":"584e4c5eb255a98e3fc9cd1a925cbdc58e88a14de21e4ade981df1a2a6512216"} Nov 02 20:15:02.330552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.330523 4966 generic.go:356] "Generic (PLEG): container finished" podID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerID="95c0718b93f5dcf1b4431d1d05f61d884a8c61eb83d90a9e2c66a4f9087710b4" exitCode=60 Nov 02 20:15:02.330832 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.330814 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" event={"ID":"5a68f258-f099-4434-bac3-e3f9f7a3bfdf","Type":"ContainerDied","Data":"95c0718b93f5dcf1b4431d1d05f61d884a8c61eb83d90a9e2c66a4f9087710b4"} Nov 02 20:15:02.330978 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.330962 4966 scope.go:122] "RemoveContainer" containerID="95c0718b93f5dcf1b4431d1d05f61d884a8c61eb83d90a9e2c66a4f9087710b4" Nov 02 20:15:02.332565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.332542 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" event={"ID":"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba","Type":"ContainerStarted","Data":"f8552b9f5a30583e787939d3a45d30479f509198242686a77196c43b863575b4"} Nov 02 20:15:02.332612 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.332583 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" event={"ID":"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba","Type":"ContainerStarted","Data":"db3618b89483f787342f200e4c0c2140b2056ae0db66a3d66f4b4eb25ae90247"} Nov 02 20:15:02.334682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.334661 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" event={"ID":"331eb841-ecb1-4fca-813e-842e3e9f0529","Type":"ContainerStarted","Data":"78ded8131fd4dbf2ea4457dea1600556f373e125b08ce84cc295293ff82b3a48"} Nov 02 20:15:02.394045 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.393987 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" podStartSLOduration=2.393967933 podStartE2EDuration="2.393967933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.388984967 +0000 UTC m=+2808.337640806" watchObservedRunningTime="2026-11-02 20:15:02.393967933 +0000 UTC m=+2808.342623780" Nov 02 20:15:02.494643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.494590 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" podStartSLOduration=2.49457279 podStartE2EDuration="2.49457279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.494493542 +0000 UTC m=+2808.443149381" watchObservedRunningTime="2026-11-02 20:15:02.49457279 +0000 UTC m=+2808.443228630" Nov 02 20:15:02.704378 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.704269 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" podStartSLOduration=2.70425574 podStartE2EDuration="2.70425574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.702641258 +0000 UTC m=+2808.651297078" watchObservedRunningTime="2026-11-02 20:15:02.70425574 +0000 UTC m=+2808.652911576" Nov 02 20:15:02.705663 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.705623 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" podStartSLOduration=2.705611714 podStartE2EDuration="2.705611714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.598749798 +0000 UTC m=+2808.547405637" watchObservedRunningTime="2026-11-02 20:15:02.705611714 +0000 UTC m=+2808.654267598" Nov 02 20:15:02.809000 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.808941 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" podStartSLOduration=2.808925608 podStartE2EDuration="2.808925608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.806289989 +0000 UTC m=+2808.754945828" watchObservedRunningTime="2026-11-02 20:15:02.808925608 +0000 UTC m=+2808.757581443" Nov 02 20:15:02.913827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:02.913746 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" podStartSLOduration=2.9137285029999997 podStartE2EDuration="2.913728503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.911198955 +0000 UTC m=+2808.859854797" watchObservedRunningTime="2026-11-02 20:15:02.913728503 +0000 UTC m=+2808.862384343" Nov 02 20:15:03.018919 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.018870 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" podStartSLOduration=3.01885507 podStartE2EDuration="3.01885507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.01779947 +0000 UTC m=+2808.966455306" watchObservedRunningTime="2026-11-02 20:15:03.01885507 +0000 UTC m=+2808.967510906" Nov 02 20:15:03.234179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.234110 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" podStartSLOduration=3.234090894 podStartE2EDuration="3.234090894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.22970107 +0000 UTC m=+2809.178356909" watchObservedRunningTime="2026-11-02 20:15:03.234090894 +0000 UTC m=+2809.182746734" Nov 02 20:15:03.348471 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.348440 4966 generic.go:356] "Generic (PLEG): container finished" podID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerID="1e674b05f00f6e87751e74ae48de898cf33bb9f45d8f0026f0c7867e77277845" exitCode=60 Nov 02 20:15:03.348611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.348529 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" event={"ID":"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1","Type":"ContainerDied","Data":"1e674b05f00f6e87751e74ae48de898cf33bb9f45d8f0026f0c7867e77277845"} Nov 02 20:15:03.348829 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.348807 4966 scope.go:122] "RemoveContainer" containerID="1e674b05f00f6e87751e74ae48de898cf33bb9f45d8f0026f0c7867e77277845" Nov 02 20:15:03.350514 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.350492 4966 generic.go:356] "Generic (PLEG): container finished" podID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerID="c9e4cfc21f6e00f783a83253cc4f670e96ea64b35a7b14d5d8a3022c2fe3553e" exitCode=60 Nov 02 20:15:03.350636 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.350568 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" event={"ID":"5addd05e-f0bb-40aa-a86c-dce2c22129fb","Type":"ContainerDied","Data":"c9e4cfc21f6e00f783a83253cc4f670e96ea64b35a7b14d5d8a3022c2fe3553e"} Nov 02 20:15:03.350826 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.350807 4966 scope.go:122] "RemoveContainer" containerID="c9e4cfc21f6e00f783a83253cc4f670e96ea64b35a7b14d5d8a3022c2fe3553e" Nov 02 20:15:03.352598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.352564 4966 generic.go:356] "Generic (PLEG): container finished" podID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerID="584e4c5eb255a98e3fc9cd1a925cbdc58e88a14de21e4ade981df1a2a6512216" exitCode=60 Nov 02 20:15:03.352664 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.352624 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerDied","Data":"584e4c5eb255a98e3fc9cd1a925cbdc58e88a14de21e4ade981df1a2a6512216"} Nov 02 20:15:03.352884 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.352865 4966 scope.go:122] "RemoveContainer" containerID="584e4c5eb255a98e3fc9cd1a925cbdc58e88a14de21e4ade981df1a2a6512216" Nov 02 20:15:03.355548 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.355474 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" event={"ID":"5a68f258-f099-4434-bac3-e3f9f7a3bfdf","Type":"ContainerStarted","Data":"5fafd583c8d22a9254be8c495df6805dc989f1d8ec2b765988763619394c021d"} Nov 02 20:15:03.357372 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.357346 4966 generic.go:356] "Generic (PLEG): container finished" podID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerID="f8552b9f5a30583e787939d3a45d30479f509198242686a77196c43b863575b4" exitCode=60 Nov 02 20:15:03.357479 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.357427 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" event={"ID":"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba","Type":"ContainerDied","Data":"f8552b9f5a30583e787939d3a45d30479f509198242686a77196c43b863575b4"} Nov 02 20:15:03.357634 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.357617 4966 scope.go:122] "RemoveContainer" containerID="f8552b9f5a30583e787939d3a45d30479f509198242686a77196c43b863575b4" Nov 02 20:15:03.359288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.359263 4966 generic.go:356] "Generic (PLEG): container finished" podID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerID="78ded8131fd4dbf2ea4457dea1600556f373e125b08ce84cc295293ff82b3a48" exitCode=60 Nov 02 20:15:03.359400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.359348 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" event={"ID":"331eb841-ecb1-4fca-813e-842e3e9f0529","Type":"ContainerDied","Data":"78ded8131fd4dbf2ea4457dea1600556f373e125b08ce84cc295293ff82b3a48"} Nov 02 20:15:03.359631 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.359613 4966 scope.go:122] "RemoveContainer" containerID="78ded8131fd4dbf2ea4457dea1600556f373e125b08ce84cc295293ff82b3a48" Nov 02 20:15:03.362278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.362254 4966 generic.go:356] "Generic (PLEG): container finished" podID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerID="f72567e987d0f25bc8c605867c6491e5f1e3af19a6556975b0141db94a8d1f27" exitCode=60 Nov 02 20:15:03.362424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.362352 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerDied","Data":"f72567e987d0f25bc8c605867c6491e5f1e3af19a6556975b0141db94a8d1f27"} Nov 02 20:15:03.362559 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.362539 4966 scope.go:122] "RemoveContainer" containerID="f72567e987d0f25bc8c605867c6491e5f1e3af19a6556975b0141db94a8d1f27" Nov 02 20:15:03.364824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.364800 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" event={"ID":"f74b7ad1-f775-4370-bb1a-4d71643eaf6f","Type":"ContainerStarted","Data":"95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5"} Nov 02 20:15:03.365143 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.365125 4966 scope.go:122] "RemoveContainer" containerID="95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5" Nov 02 20:15:03.365405 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:03.365379 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm_odh-ai-gateway-infra(f74b7ad1-f775-4370-bb1a-4d71643eaf6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" Nov 02 20:15:03.368040 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.367995 4966 generic.go:356] "Generic (PLEG): container finished" podID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerID="299fcb8d6a891b50b45507909cf4642d985cf2453d4f0f51ac902b961bcfbed3" exitCode=60 Nov 02 20:15:03.369026 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.368848 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" event={"ID":"d3a3287b-f705-4b73-aed1-5c04e4febbc2","Type":"ContainerDied","Data":"299fcb8d6a891b50b45507909cf4642d985cf2453d4f0f51ac902b961bcfbed3"} Nov 02 20:15:03.371083 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.371066 4966 scope.go:122] "RemoveContainer" containerID="299fcb8d6a891b50b45507909cf4642d985cf2453d4f0f51ac902b961bcfbed3" Nov 02 20:15:03.375656 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.375632 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" event={"ID":"d04eecc2-cc32-4536-8a4b-de0b553306c1","Type":"ContainerStarted","Data":"c51c1f604e7c4905eec9e49c07082e40b451a7fccc146dea42d3888632613562"} Nov 02 20:15:03.381261 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.380595 4966 generic.go:356] "Generic (PLEG): container finished" podID="a825edf8-6b57-40d8-81a4-873300069d62" containerID="e02e424a5ea631eef6366824eb366f4b7deacf3231e1b88fd146f6f54689b970" exitCode=60 Nov 02 20:15:03.381261 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.380635 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" event={"ID":"a825edf8-6b57-40d8-81a4-873300069d62","Type":"ContainerDied","Data":"e02e424a5ea631eef6366824eb366f4b7deacf3231e1b88fd146f6f54689b970"} Nov 02 20:15:03.381261 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.380915 4966 scope.go:122] "RemoveContainer" containerID="e02e424a5ea631eef6366824eb366f4b7deacf3231e1b88fd146f6f54689b970" Nov 02 20:15:03.591630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.591573 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" podStartSLOduration=3.591550322 podStartE2EDuration="3.591550322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.590816111 +0000 UTC m=+2809.539471949" watchObservedRunningTime="2026-11-02 20:15:03.591550322 +0000 UTC m=+2809.540206164" Nov 02 20:15:03.904739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:03.904671 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" podStartSLOduration=3.904655247 podStartE2EDuration="3.904655247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.903726448 +0000 UTC m=+2809.852382286" watchObservedRunningTime="2026-11-02 20:15:03.904655247 +0000 UTC m=+2809.853311086" Nov 02 20:15:04.392283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.392192 4966 generic.go:356] "Generic (PLEG): container finished" podID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerID="c51c1f604e7c4905eec9e49c07082e40b451a7fccc146dea42d3888632613562" exitCode=60 Nov 02 20:15:04.392283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.392266 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" event={"ID":"d04eecc2-cc32-4536-8a4b-de0b553306c1","Type":"ContainerDied","Data":"c51c1f604e7c4905eec9e49c07082e40b451a7fccc146dea42d3888632613562"} Nov 02 20:15:04.392542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.392367 4966 scope.go:122] "RemoveContainer" containerID="ad8eb08593a7710407b00f22c536976d7464461bb1be9749d2f9a472cd5b6158" Nov 02 20:15:04.392673 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.392656 4966 scope.go:122] "RemoveContainer" containerID="c51c1f604e7c4905eec9e49c07082e40b451a7fccc146dea42d3888632613562" Nov 02 20:15:04.393006 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:04.392977 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n_odh-ai-gateway-infra(d04eecc2-cc32-4536-8a4b-de0b553306c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" Nov 02 20:15:04.394377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.394346 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" event={"ID":"a825edf8-6b57-40d8-81a4-873300069d62","Type":"ContainerStarted","Data":"82bc5aacc2a5ca469893b2ef007a4643641e60fbc7ca63b7a76308c1f9a60884"} Nov 02 20:15:04.396599 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.396575 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" event={"ID":"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1","Type":"ContainerStarted","Data":"7d31c58ae0963250b25b6b3d75fb3fe60f84be9cca5c272f0dc04b2d0fe782d6"} Nov 02 20:15:04.399537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.399512 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" event={"ID":"5addd05e-f0bb-40aa-a86c-dce2c22129fb","Type":"ContainerStarted","Data":"6bc99c7ee1941f999b78c06d5de85841e36d1ca9b93373b601cdc5024c15fb88"} Nov 02 20:15:04.401936 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.401908 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerStarted","Data":"d06f031753257b508718cfd53654ac32d8ccd1d3854e04698aaa813d6b13be99"} Nov 02 20:15:04.403931 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.403908 4966 generic.go:356] "Generic (PLEG): container finished" podID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerID="5fafd583c8d22a9254be8c495df6805dc989f1d8ec2b765988763619394c021d" exitCode=60 Nov 02 20:15:04.404030 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.403988 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" event={"ID":"5a68f258-f099-4434-bac3-e3f9f7a3bfdf","Type":"ContainerDied","Data":"5fafd583c8d22a9254be8c495df6805dc989f1d8ec2b765988763619394c021d"} Nov 02 20:15:04.404169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.404152 4966 scope.go:122] "RemoveContainer" containerID="5fafd583c8d22a9254be8c495df6805dc989f1d8ec2b765988763619394c021d" Nov 02 20:15:04.404429 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:04.404407 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr_odh-ai-gateway-infra(5a68f258-f099-4434-bac3-e3f9f7a3bfdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" Nov 02 20:15:04.406101 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.406065 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" event={"ID":"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba","Type":"ContainerStarted","Data":"e262bc6c5d4a48c5547b3d3f9d165861857e77217ac97e8c567a928d0f9d316e"} Nov 02 20:15:04.407363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.407343 4966 scope.go:122] "RemoveContainer" containerID="95c0718b93f5dcf1b4431d1d05f61d884a8c61eb83d90a9e2c66a4f9087710b4" Nov 02 20:15:04.408990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.408968 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" event={"ID":"331eb841-ecb1-4fca-813e-842e3e9f0529","Type":"ContainerStarted","Data":"f0b4b4ae647df7e15f35fafbd29492cf5cff4b4e2fcf44bbb7f7e29e05790984"} Nov 02 20:15:04.410751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.410730 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerStarted","Data":"be6a6c64ab508eb7a98bbe51e86d9449101206df5179d99f7e93e03ea38c8e02"} Nov 02 20:15:04.412533 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.412509 4966 generic.go:356] "Generic (PLEG): container finished" podID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerID="95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5" exitCode=60 Nov 02 20:15:04.412640 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.412584 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" event={"ID":"f74b7ad1-f775-4370-bb1a-4d71643eaf6f","Type":"ContainerDied","Data":"95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5"} Nov 02 20:15:04.412842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.412825 4966 scope.go:122] "RemoveContainer" containerID="95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5" Nov 02 20:15:04.413113 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:04.413088 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm_odh-ai-gateway-infra(f74b7ad1-f775-4370-bb1a-4d71643eaf6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" Nov 02 20:15:04.414400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.414379 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" event={"ID":"d3a3287b-f705-4b73-aed1-5c04e4febbc2","Type":"ContainerStarted","Data":"4507d83f4eb850cb6c9c944e2527cbf720b5115521366553474d1cbb60f56b37"} Nov 02 20:15:04.422006 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:04.421987 4966 scope.go:122] "RemoveContainer" containerID="d516dd63c2a360e006df0580d304238273a89e62f0816586e6dcd67b6ddda497" Nov 02 20:15:05.425071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.425032 4966 generic.go:356] "Generic (PLEG): container finished" podID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerID="7d31c58ae0963250b25b6b3d75fb3fe60f84be9cca5c272f0dc04b2d0fe782d6" exitCode=60 Nov 02 20:15:05.425606 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.425113 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" event={"ID":"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1","Type":"ContainerDied","Data":"7d31c58ae0963250b25b6b3d75fb3fe60f84be9cca5c272f0dc04b2d0fe782d6"} Nov 02 20:15:05.425606 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.425176 4966 scope.go:122] "RemoveContainer" containerID="1e674b05f00f6e87751e74ae48de898cf33bb9f45d8f0026f0c7867e77277845" Nov 02 20:15:05.425606 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.425496 4966 scope.go:122] "RemoveContainer" containerID="7d31c58ae0963250b25b6b3d75fb3fe60f84be9cca5c272f0dc04b2d0fe782d6" Nov 02 20:15:05.425885 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.425849 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9_odh-ai-gateway-infra(971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" Nov 02 20:15:05.427279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.427254 4966 generic.go:356] "Generic (PLEG): container finished" podID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerID="6bc99c7ee1941f999b78c06d5de85841e36d1ca9b93373b601cdc5024c15fb88" exitCode=60 Nov 02 20:15:05.427413 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.427327 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" event={"ID":"5addd05e-f0bb-40aa-a86c-dce2c22129fb","Type":"ContainerDied","Data":"6bc99c7ee1941f999b78c06d5de85841e36d1ca9b93373b601cdc5024c15fb88"} Nov 02 20:15:05.427608 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.427591 4966 scope.go:122] "RemoveContainer" containerID="6bc99c7ee1941f999b78c06d5de85841e36d1ca9b93373b601cdc5024c15fb88" Nov 02 20:15:05.427898 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.427879 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w_odh-ai-gateway-infra(5addd05e-f0bb-40aa-a86c-dce2c22129fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" Nov 02 20:15:05.429249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.429232 4966 generic.go:356] "Generic (PLEG): container finished" podID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerID="d06f031753257b508718cfd53654ac32d8ccd1d3854e04698aaa813d6b13be99" exitCode=60 Nov 02 20:15:05.429339 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.429289 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerDied","Data":"d06f031753257b508718cfd53654ac32d8ccd1d3854e04698aaa813d6b13be99"} Nov 02 20:15:05.429557 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.429542 4966 scope.go:122] "RemoveContainer" containerID="d06f031753257b508718cfd53654ac32d8ccd1d3854e04698aaa813d6b13be99" Nov 02 20:15:05.429832 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.429811 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv_odh-ai-gateway-infra(4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" Nov 02 20:15:05.431226 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.431210 4966 scope.go:122] "RemoveContainer" containerID="5fafd583c8d22a9254be8c495df6805dc989f1d8ec2b765988763619394c021d" Nov 02 20:15:05.431466 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.431445 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr_odh-ai-gateway-infra(5a68f258-f099-4434-bac3-e3f9f7a3bfdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" Nov 02 20:15:05.433217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.433198 4966 generic.go:356] "Generic (PLEG): container finished" podID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerID="e262bc6c5d4a48c5547b3d3f9d165861857e77217ac97e8c567a928d0f9d316e" exitCode=60 Nov 02 20:15:05.433338 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.433268 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" event={"ID":"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba","Type":"ContainerDied","Data":"e262bc6c5d4a48c5547b3d3f9d165861857e77217ac97e8c567a928d0f9d316e"} Nov 02 20:15:05.433610 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.433591 4966 scope.go:122] "RemoveContainer" containerID="e262bc6c5d4a48c5547b3d3f9d165861857e77217ac97e8c567a928d0f9d316e" Nov 02 20:15:05.433915 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.433836 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh_odh-ai-gateway-infra(6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" Nov 02 20:15:05.435554 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.435532 4966 generic.go:356] "Generic (PLEG): container finished" podID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerID="f0b4b4ae647df7e15f35fafbd29492cf5cff4b4e2fcf44bbb7f7e29e05790984" exitCode=60 Nov 02 20:15:05.435649 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.435616 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" event={"ID":"331eb841-ecb1-4fca-813e-842e3e9f0529","Type":"ContainerDied","Data":"f0b4b4ae647df7e15f35fafbd29492cf5cff4b4e2fcf44bbb7f7e29e05790984"} Nov 02 20:15:05.435799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.435786 4966 scope.go:122] "RemoveContainer" containerID="f0b4b4ae647df7e15f35fafbd29492cf5cff4b4e2fcf44bbb7f7e29e05790984" Nov 02 20:15:05.436000 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.435977 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl_odh-ai-gateway-infra(331eb841-ecb1-4fca-813e-842e3e9f0529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" Nov 02 20:15:05.440441 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.440419 4966 scope.go:122] "RemoveContainer" containerID="c9e4cfc21f6e00f783a83253cc4f670e96ea64b35a7b14d5d8a3022c2fe3553e" Nov 02 20:15:05.440524 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.440497 4966 generic.go:356] "Generic (PLEG): container finished" podID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerID="be6a6c64ab508eb7a98bbe51e86d9449101206df5179d99f7e93e03ea38c8e02" exitCode=60 Nov 02 20:15:05.440588 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.440571 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerDied","Data":"be6a6c64ab508eb7a98bbe51e86d9449101206df5179d99f7e93e03ea38c8e02"} Nov 02 20:15:05.440761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.440744 4966 scope.go:122] "RemoveContainer" containerID="be6a6c64ab508eb7a98bbe51e86d9449101206df5179d99f7e93e03ea38c8e02" Nov 02 20:15:05.440974 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.440942 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg_odh-ai-gateway-infra(df678d54-f0bd-4b95-b86a-2c67ae313fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" Nov 02 20:15:05.442618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.442597 4966 scope.go:122] "RemoveContainer" containerID="95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5" Nov 02 20:15:05.442842 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.442811 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm_odh-ai-gateway-infra(f74b7ad1-f775-4370-bb1a-4d71643eaf6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" Nov 02 20:15:05.444382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.444291 4966 generic.go:356] "Generic (PLEG): container finished" podID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerID="4507d83f4eb850cb6c9c944e2527cbf720b5115521366553474d1cbb60f56b37" exitCode=60 Nov 02 20:15:05.444490 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.444405 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" event={"ID":"d3a3287b-f705-4b73-aed1-5c04e4febbc2","Type":"ContainerDied","Data":"4507d83f4eb850cb6c9c944e2527cbf720b5115521366553474d1cbb60f56b37"} Nov 02 20:15:05.444578 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.444560 4966 scope.go:122] "RemoveContainer" containerID="4507d83f4eb850cb6c9c944e2527cbf720b5115521366553474d1cbb60f56b37" Nov 02 20:15:05.444770 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.444740 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr_odh-ai-gateway-infra(d3a3287b-f705-4b73-aed1-5c04e4febbc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" Nov 02 20:15:05.446518 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.446498 4966 scope.go:122] "RemoveContainer" containerID="c51c1f604e7c4905eec9e49c07082e40b451a7fccc146dea42d3888632613562" Nov 02 20:15:05.446722 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.446694 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n_odh-ai-gateway-infra(d04eecc2-cc32-4536-8a4b-de0b553306c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" Nov 02 20:15:05.448584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.448562 4966 generic.go:356] "Generic (PLEG): container finished" podID="a825edf8-6b57-40d8-81a4-873300069d62" containerID="82bc5aacc2a5ca469893b2ef007a4643641e60fbc7ca63b7a76308c1f9a60884" exitCode=60 Nov 02 20:15:05.448674 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.448607 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" event={"ID":"a825edf8-6b57-40d8-81a4-873300069d62","Type":"ContainerDied","Data":"82bc5aacc2a5ca469893b2ef007a4643641e60fbc7ca63b7a76308c1f9a60884"} Nov 02 20:15:05.448843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.448826 4966 scope.go:122] "RemoveContainer" containerID="82bc5aacc2a5ca469893b2ef007a4643641e60fbc7ca63b7a76308c1f9a60884" Nov 02 20:15:05.449052 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:05.449032 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc_odh-ai-gateway-infra(a825edf8-6b57-40d8-81a4-873300069d62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" podUID="a825edf8-6b57-40d8-81a4-873300069d62" Nov 02 20:15:05.456001 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.455971 4966 scope.go:122] "RemoveContainer" containerID="584e4c5eb255a98e3fc9cd1a925cbdc58e88a14de21e4ade981df1a2a6512216" Nov 02 20:15:05.470617 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.470591 4966 scope.go:122] "RemoveContainer" containerID="f8552b9f5a30583e787939d3a45d30479f509198242686a77196c43b863575b4" Nov 02 20:15:05.489665 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.489640 4966 scope.go:122] "RemoveContainer" containerID="78ded8131fd4dbf2ea4457dea1600556f373e125b08ce84cc295293ff82b3a48" Nov 02 20:15:05.503386 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.503366 4966 scope.go:122] "RemoveContainer" containerID="f72567e987d0f25bc8c605867c6491e5f1e3af19a6556975b0141db94a8d1f27" Nov 02 20:15:05.516437 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.516418 4966 scope.go:122] "RemoveContainer" containerID="299fcb8d6a891b50b45507909cf4642d985cf2453d4f0f51ac902b961bcfbed3" Nov 02 20:15:05.529938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.529919 4966 scope.go:122] "RemoveContainer" containerID="e02e424a5ea631eef6366824eb366f4b7deacf3231e1b88fd146f6f54689b970" Nov 02 20:15:05.726536 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.726493 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h"] Nov 02 20:15:05.727179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.727158 4966 cpu_manager.go:414] "containerMap: removing container" podUID="28cab378-908f-4877-af24-685c7d93e7ae" containerName="revoke-keys" Nov 02 20:15:05.727328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.727180 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28cab378-908f-4877-af24-685c7d93e7ae" containerName="revoke-keys" Nov 02 20:15:05.727393 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.727382 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="28cab378-908f-4877-af24-685c7d93e7ae" containerName="revoke-keys" Nov 02 20:15:05.732283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.732261 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.781146 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.781110 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:15:05.785114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.785084 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h"] Nov 02 20:15:05.851001 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.850953 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xt67\" (UniqueName: \"kubernetes.io/projected/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-kube-api-access-2xt67\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.851222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.851201 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.851313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.851240 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.952528 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.952493 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.952528 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.952531 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.952806 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.952567 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xt67\" (UniqueName: \"kubernetes.io/projected/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-kube-api-access-2xt67\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.953291 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.953267 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:05.955017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:05.954987 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:06.006066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.005988 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xt67\" (UniqueName: \"kubernetes.io/projected/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-kube-api-access-2xt67\") pod \"maas-api-revoke-sub-281fe165-66c4h\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:06.047020 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.046976 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:06.434911 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.434872 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h"] Nov 02 20:15:06.435722 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:06.435694 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d190c1_32e8_4b22_9b4e_0eedcba467d9.slice/crio-9589b0030228b974e408752f911a2194fe664037bd188eef746549123d38a464 WatchSource:0}: Error finding container 9589b0030228b974e408752f911a2194fe664037bd188eef746549123d38a464: Status 404 returned error can't find the container with id 9589b0030228b974e408752f911a2194fe664037bd188eef746549123d38a464 Nov 02 20:15:06.460254 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.460225 4966 scope.go:122] "RemoveContainer" containerID="7d31c58ae0963250b25b6b3d75fb3fe60f84be9cca5c272f0dc04b2d0fe782d6" Nov 02 20:15:06.460574 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.460540 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9_odh-ai-gateway-infra(971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" Nov 02 20:15:06.461380 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.461350 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" event={"ID":"b4d190c1-32e8-4b22-9b4e-0eedcba467d9","Type":"ContainerStarted","Data":"9589b0030228b974e408752f911a2194fe664037bd188eef746549123d38a464"} Nov 02 20:15:06.463716 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.463698 4966 scope.go:122] "RemoveContainer" containerID="6bc99c7ee1941f999b78c06d5de85841e36d1ca9b93373b601cdc5024c15fb88" Nov 02 20:15:06.463984 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.463960 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w_odh-ai-gateway-infra(5addd05e-f0bb-40aa-a86c-dce2c22129fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" Nov 02 20:15:06.473970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.473934 4966 scope.go:122] "RemoveContainer" containerID="d06f031753257b508718cfd53654ac32d8ccd1d3854e04698aaa813d6b13be99" Nov 02 20:15:06.474239 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.474205 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv_odh-ai-gateway-infra(4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" Nov 02 20:15:06.476340 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.476288 4966 scope.go:122] "RemoveContainer" containerID="e262bc6c5d4a48c5547b3d3f9d165861857e77217ac97e8c567a928d0f9d316e" Nov 02 20:15:06.476609 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.476584 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh_odh-ai-gateway-infra(6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" Nov 02 20:15:06.478232 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.478214 4966 scope.go:122] "RemoveContainer" containerID="f0b4b4ae647df7e15f35fafbd29492cf5cff4b4e2fcf44bbb7f7e29e05790984" Nov 02 20:15:06.478466 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.478445 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl_odh-ai-gateway-infra(331eb841-ecb1-4fca-813e-842e3e9f0529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" Nov 02 20:15:06.480029 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.480011 4966 scope.go:122] "RemoveContainer" containerID="be6a6c64ab508eb7a98bbe51e86d9449101206df5179d99f7e93e03ea38c8e02" Nov 02 20:15:06.480226 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.480204 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg_odh-ai-gateway-infra(df678d54-f0bd-4b95-b86a-2c67ae313fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" Nov 02 20:15:06.481679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.481661 4966 scope.go:122] "RemoveContainer" containerID="4507d83f4eb850cb6c9c944e2527cbf720b5115521366553474d1cbb60f56b37" Nov 02 20:15:06.481902 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.481877 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr_odh-ai-gateway-infra(d3a3287b-f705-4b73-aed1-5c04e4febbc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" Nov 02 20:15:06.483523 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:06.483502 4966 scope.go:122] "RemoveContainer" containerID="82bc5aacc2a5ca469893b2ef007a4643641e60fbc7ca63b7a76308c1f9a60884" Nov 02 20:15:06.483742 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:06.483720 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc_odh-ai-gateway-infra(a825edf8-6b57-40d8-81a4-873300069d62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" podUID="a825edf8-6b57-40d8-81a4-873300069d62" Nov 02 20:15:07.494892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:07.494852 4966 generic.go:356] "Generic (PLEG): container finished" podID="b4d190c1-32e8-4b22-9b4e-0eedcba467d9" containerID="11337a81eb390fa7d917ad70acee0e5a0f1b7d6f2c7d0bfb154dc575aeeb96f2" exitCode=0 Nov 02 20:15:07.495252 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:07.494886 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" event={"ID":"b4d190c1-32e8-4b22-9b4e-0eedcba467d9","Type":"ContainerDied","Data":"11337a81eb390fa7d917ad70acee0e5a0f1b7d6f2c7d0bfb154dc575aeeb96f2"} Nov 02 20:15:08.660684 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.660658 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:08.784400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.784279 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-openshift-ca\") pod \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " Nov 02 20:15:08.784587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.784535 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-xks-ca\") pod \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " Nov 02 20:15:08.784665 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.784585 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-kube-api-access-2xt67\" (UniqueName: \"kubernetes.io/projected/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-kube-api-access-2xt67\") pod \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\" (UID: \"b4d190c1-32e8-4b22-9b4e-0eedcba467d9\") " Nov 02 20:15:08.784733 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.784696 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-openshift-ca" pod "b4d190c1-32e8-4b22-9b4e-0eedcba467d9" (UID: "b4d190c1-32e8-4b22-9b4e-0eedcba467d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:08.785054 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.785025 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:08.787056 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.787031 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-xks-ca" pod "b4d190c1-32e8-4b22-9b4e-0eedcba467d9" (UID: "b4d190c1-32e8-4b22-9b4e-0eedcba467d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:08.788033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.787546 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-kube-api-access-2xt67" pod "b4d190c1-32e8-4b22-9b4e-0eedcba467d9" (UID: "b4d190c1-32e8-4b22-9b4e-0eedcba467d9"). InnerVolumeSpecName "kube-api-access-2xt67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:08.885809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.885757 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:08.885809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:08.885801 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xt67\" (UniqueName: \"kubernetes.io/projected/b4d190c1-32e8-4b22-9b4e-0eedcba467d9-kube-api-access-2xt67\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:09.513391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:09.513361 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" Nov 02 20:15:09.513566 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:09.513394 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h" event={"ID":"b4d190c1-32e8-4b22-9b4e-0eedcba467d9","Type":"ContainerDied","Data":"9589b0030228b974e408752f911a2194fe664037bd188eef746549123d38a464"} Nov 02 20:15:09.513566 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:09.513442 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9589b0030228b974e408752f911a2194fe664037bd188eef746549123d38a464" Nov 02 20:15:10.795092 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:10.795059 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9"] Nov 02 20:15:10.999269 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:10.999244 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:11.112648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.112549 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-kube-api-access-9xdpf\" (UniqueName: \"kubernetes.io/projected/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-kube-api-access-9xdpf\") pod \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " Nov 02 20:15:11.112648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.112620 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-maas-api-service-ca\") pod \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\" (UID: \"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1\") " Nov 02 20:15:11.113379 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.113345 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-maas-api-service-ca" pod "971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" (UID: "971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:11.115779 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.115737 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-kube-api-access-9xdpf" pod "971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" (UID: "971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1"). InnerVolumeSpecName "kube-api-access-9xdpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:11.214469 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.214427 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:11.214469 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.214462 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xdpf\" (UniqueName: \"kubernetes.io/projected/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1-kube-api-access-9xdpf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:11.530992 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.530952 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" Nov 02 20:15:11.531214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.530988 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9" event={"ID":"971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1","Type":"ContainerDied","Data":"6bd327167a85ceb5879021e7c19c946481ca9718037f8bf507e6311acdd7e89c"} Nov 02 20:15:11.531214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.531051 4966 scope.go:122] "RemoveContainer" containerID="7d31c58ae0963250b25b6b3d75fb3fe60f84be9cca5c272f0dc04b2d0fe782d6" Nov 02 20:15:11.684198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.684161 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9"] Nov 02 20:15:11.687663 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:11.687637 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b45a2b-29894175-kjcn9"] Nov 02 20:15:12.183340 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.183292 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" path="/var/lib/kubelet/pods/971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1/volumes" Nov 02 20:15:12.301382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.301343 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" podUID="e5bf48b0-4073-420d-83b0-4f577342622e" containerName="main" containerID="cri-o://8ea133c3c27f6ec129f88bdaeaf8f8d068f607c89cce95771bc217bc0b8a34c0" gracePeriod=45 Nov 02 20:15:12.540825 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.540789 4966 generic.go:356] "Generic (PLEG): container finished" podID="e5bf48b0-4073-420d-83b0-4f577342622e" containerID="8ea133c3c27f6ec129f88bdaeaf8f8d068f607c89cce95771bc217bc0b8a34c0" exitCode=0 Nov 02 20:15:12.540996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.540866 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" event={"ID":"e5bf48b0-4073-420d-83b0-4f577342622e","Type":"ContainerDied","Data":"8ea133c3c27f6ec129f88bdaeaf8f8d068f607c89cce95771bc217bc0b8a34c0"} Nov 02 20:15:12.581896 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.581868 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:15:12.680404 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.680361 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs"] Nov 02 20:15:12.684115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.684088 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b45a2b-sx7cs"] Nov 02 20:15:12.730766 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.730729 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-dshm\") pod \"e5bf48b0-4073-420d-83b0-4f577342622e\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " Nov 02 20:15:12.730947 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.730926 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5bf48b0-4073-420d-83b0-4f577342622e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5bf48b0-4073-420d-83b0-4f577342622e-tls-certs\") pod \"e5bf48b0-4073-420d-83b0-4f577342622e\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " Nov 02 20:15:12.730987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.730959 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-model-cache\") pod \"e5bf48b0-4073-420d-83b0-4f577342622e\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " Nov 02 20:15:12.731033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.730989 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bf48b0-4073-420d-83b0-4f577342622e-kube-api-access-8846p\" (UniqueName: \"kubernetes.io/projected/e5bf48b0-4073-420d-83b0-4f577342622e-kube-api-access-8846p\") pod \"e5bf48b0-4073-420d-83b0-4f577342622e\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " Nov 02 20:15:12.731122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.731062 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-home\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-home\") pod \"e5bf48b0-4073-420d-83b0-4f577342622e\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " Nov 02 20:15:12.731122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.731100 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-tmp-dir\") pod \"e5bf48b0-4073-420d-83b0-4f577342622e\" (UID: \"e5bf48b0-4073-420d-83b0-4f577342622e\") " Nov 02 20:15:12.731558 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.731494 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-home" pod "e5bf48b0-4073-420d-83b0-4f577342622e" (UID: "e5bf48b0-4073-420d-83b0-4f577342622e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:12.731722 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.731666 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-tmp-dir" pod "e5bf48b0-4073-420d-83b0-4f577342622e" (UID: "e5bf48b0-4073-420d-83b0-4f577342622e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:12.731831 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.731809 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-model-cache" pod "e5bf48b0-4073-420d-83b0-4f577342622e" (UID: "e5bf48b0-4073-420d-83b0-4f577342622e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:12.733474 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.733449 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-dshm" pod "e5bf48b0-4073-420d-83b0-4f577342622e" (UID: "e5bf48b0-4073-420d-83b0-4f577342622e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:12.733578 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.733558 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5bf48b0-4073-420d-83b0-4f577342622e-tls-certs" pod "e5bf48b0-4073-420d-83b0-4f577342622e" (UID: "e5bf48b0-4073-420d-83b0-4f577342622e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:12.733638 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.733617 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bf48b0-4073-420d-83b0-4f577342622e-kube-api-access-8846p" pod "e5bf48b0-4073-420d-83b0-4f577342622e" (UID: "e5bf48b0-4073-420d-83b0-4f577342622e"). InnerVolumeSpecName "kube-api-access-8846p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:12.832804 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.832771 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5bf48b0-4073-420d-83b0-4f577342622e-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:12.832804 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.832802 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:12.832804 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.832814 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8846p\" (UniqueName: \"kubernetes.io/projected/e5bf48b0-4073-420d-83b0-4f577342622e-kube-api-access-8846p\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:12.833066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.832824 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:12.833066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.832835 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:12.833066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:12.832843 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5bf48b0-4073-420d-83b0-4f577342622e-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:13.549849 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:13.549810 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" event={"ID":"e5bf48b0-4073-420d-83b0-4f577342622e","Type":"ContainerDied","Data":"0b69ea997e1c98277b1d5d0dd90933069e152ead241b1f03ddcf66006cf3965c"} Nov 02 20:15:13.550396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:13.549856 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6" Nov 02 20:15:13.550396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:13.549873 4966 scope.go:122] "RemoveContainer" containerID="8ea133c3c27f6ec129f88bdaeaf8f8d068f607c89cce95771bc217bc0b8a34c0" Nov 02 20:15:13.706249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:13.706213 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6"] Nov 02 20:15:13.752256 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:13.752219 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-19d134/test-model-19d134-kserve-8595896f7f-xcqt6"] Nov 02 20:15:14.185467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:14.185434 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bf48b0-4073-420d-83b0-4f577342622e" path="/var/lib/kubelet/pods/e5bf48b0-4073-420d-83b0-4f577342622e/volumes" Nov 02 20:15:14.186408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:14.186384 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9b3a2d-d165-48b5-b31c-5f33ab80d70e" path="/var/lib/kubelet/pods/ea9b3a2d-d165-48b5-b31c-5f33ab80d70e/volumes" Nov 02 20:15:14.187226 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:14.187208 4966 scope.go:122] "RemoveContainer" containerID="95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5" Nov 02 20:15:14.187362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:14.187346 4966 scope.go:122] "RemoveContainer" containerID="5fafd583c8d22a9254be8c495df6805dc989f1d8ec2b765988763619394c021d" Nov 02 20:15:14.755773 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:14.755733 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:15:15.176105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.176063 4966 scope.go:122] "RemoveContainer" containerID="c51c1f604e7c4905eec9e49c07082e40b451a7fccc146dea42d3888632613562" Nov 02 20:15:15.176332 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.176293 4966 scope.go:122] "RemoveContainer" containerID="6bc99c7ee1941f999b78c06d5de85841e36d1ca9b93373b601cdc5024c15fb88" Nov 02 20:15:15.176504 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.176480 4966 scope.go:122] "RemoveContainer" containerID="f0b4b4ae647df7e15f35fafbd29492cf5cff4b4e2fcf44bbb7f7e29e05790984" Nov 02 20:15:15.573087 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.573042 4966 generic.go:356] "Generic (PLEG): container finished" podID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerID="1ea567fc119771badc0fc6f8c4224056d8fb589bcf4b4a480ad9d96cdc66be40" exitCode=60 Nov 02 20:15:15.573285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.573120 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" event={"ID":"5a68f258-f099-4434-bac3-e3f9f7a3bfdf","Type":"ContainerDied","Data":"1ea567fc119771badc0fc6f8c4224056d8fb589bcf4b4a480ad9d96cdc66be40"} Nov 02 20:15:15.573285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.573197 4966 scope.go:122] "RemoveContainer" containerID="5fafd583c8d22a9254be8c495df6805dc989f1d8ec2b765988763619394c021d" Nov 02 20:15:15.573608 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.573581 4966 scope.go:122] "RemoveContainer" containerID="1ea567fc119771badc0fc6f8c4224056d8fb589bcf4b4a480ad9d96cdc66be40" Nov 02 20:15:15.573914 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:15.573893 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr_odh-ai-gateway-infra(5a68f258-f099-4434-bac3-e3f9f7a3bfdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" Nov 02 20:15:15.575889 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.575867 4966 generic.go:356] "Generic (PLEG): container finished" podID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerID="7333f6a0491b6898a8f7d93d057b931afab2bef92b63a1a6973e96c5934baf5b" exitCode=60 Nov 02 20:15:15.576009 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.575944 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" event={"ID":"f74b7ad1-f775-4370-bb1a-4d71643eaf6f","Type":"ContainerDied","Data":"7333f6a0491b6898a8f7d93d057b931afab2bef92b63a1a6973e96c5934baf5b"} Nov 02 20:15:15.576252 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.576237 4966 scope.go:122] "RemoveContainer" containerID="7333f6a0491b6898a8f7d93d057b931afab2bef92b63a1a6973e96c5934baf5b" Nov 02 20:15:15.576480 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:15.576461 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm_odh-ai-gateway-infra(f74b7ad1-f775-4370-bb1a-4d71643eaf6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" Nov 02 20:15:15.590446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:15.590425 4966 scope.go:122] "RemoveContainer" containerID="95a8fa79978aeee9c5c68916bbf0e3c159e6298791acf2767ca5decb5887a1b5" Nov 02 20:15:16.176457 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.176423 4966 scope.go:122] "RemoveContainer" containerID="be6a6c64ab508eb7a98bbe51e86d9449101206df5179d99f7e93e03ea38c8e02" Nov 02 20:15:16.176907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.176626 4966 scope.go:122] "RemoveContainer" containerID="d06f031753257b508718cfd53654ac32d8ccd1d3854e04698aaa813d6b13be99" Nov 02 20:15:16.585313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.585257 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerStarted","Data":"ce1fd17b32cefb3c86d2e362cac4402de2dc075414719d4db30ed304a31385dd"} Nov 02 20:15:16.588999 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.588969 4966 generic.go:356] "Generic (PLEG): container finished" podID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerID="fd7d00716f6b02f795a23f025a29e471b4b4bd54da3e177539c9d06a463d616a" exitCode=60 Nov 02 20:15:16.589128 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.589035 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" event={"ID":"d04eecc2-cc32-4536-8a4b-de0b553306c1","Type":"ContainerDied","Data":"fd7d00716f6b02f795a23f025a29e471b4b4bd54da3e177539c9d06a463d616a"} Nov 02 20:15:16.589128 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.589085 4966 scope.go:122] "RemoveContainer" containerID="c51c1f604e7c4905eec9e49c07082e40b451a7fccc146dea42d3888632613562" Nov 02 20:15:16.589435 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.589417 4966 scope.go:122] "RemoveContainer" containerID="fd7d00716f6b02f795a23f025a29e471b4b4bd54da3e177539c9d06a463d616a" Nov 02 20:15:16.589753 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:16.589723 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n_odh-ai-gateway-infra(d04eecc2-cc32-4536-8a4b-de0b553306c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" Nov 02 20:15:16.591159 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.591140 4966 generic.go:356] "Generic (PLEG): container finished" podID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerID="40188ca4cca3c10e2a3c8024390cb275eca021c5a7a67b7ba683dcc3389237d9" exitCode=60 Nov 02 20:15:16.591258 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.591189 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" event={"ID":"5addd05e-f0bb-40aa-a86c-dce2c22129fb","Type":"ContainerDied","Data":"40188ca4cca3c10e2a3c8024390cb275eca021c5a7a67b7ba683dcc3389237d9"} Nov 02 20:15:16.591547 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.591526 4966 scope.go:122] "RemoveContainer" containerID="40188ca4cca3c10e2a3c8024390cb275eca021c5a7a67b7ba683dcc3389237d9" Nov 02 20:15:16.591790 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:16.591773 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w_odh-ai-gateway-infra(5addd05e-f0bb-40aa-a86c-dce2c22129fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" Nov 02 20:15:16.593089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.593067 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerStarted","Data":"00c550698d3aafcf51f7301713101f308123782941a2b0dbdbc5747fe7bedbd0"} Nov 02 20:15:16.596539 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.596519 4966 generic.go:356] "Generic (PLEG): container finished" podID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerID="0af4d6aec581de3f1c79800f3ab16b8008eba7d18a0a07837a6d3b97b5a0333f" exitCode=60 Nov 02 20:15:16.596632 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.596585 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" event={"ID":"331eb841-ecb1-4fca-813e-842e3e9f0529","Type":"ContainerDied","Data":"0af4d6aec581de3f1c79800f3ab16b8008eba7d18a0a07837a6d3b97b5a0333f"} Nov 02 20:15:16.596856 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.596839 4966 scope.go:122] "RemoveContainer" containerID="0af4d6aec581de3f1c79800f3ab16b8008eba7d18a0a07837a6d3b97b5a0333f" Nov 02 20:15:16.597048 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:16.597032 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl_odh-ai-gateway-infra(331eb841-ecb1-4fca-813e-842e3e9f0529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" Nov 02 20:15:16.602222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.602207 4966 scope.go:122] "RemoveContainer" containerID="6bc99c7ee1941f999b78c06d5de85841e36d1ca9b93373b601cdc5024c15fb88" Nov 02 20:15:16.615963 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.615937 4966 scope.go:122] "RemoveContainer" containerID="f0b4b4ae647df7e15f35fafbd29492cf5cff4b4e2fcf44bbb7f7e29e05790984" Nov 02 20:15:16.689436 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.689393 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr"] Nov 02 20:15:16.796808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.796751 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm"] Nov 02 20:15:16.850429 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.850400 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:16.944665 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.944607 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:16.980648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.980611 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-maas-api-service-ca\") pod \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " Nov 02 20:15:16.980828 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.980788 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-kube-api-access-zffk6\" (UniqueName: \"kubernetes.io/projected/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-kube-api-access-zffk6\") pod \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\" (UID: \"5a68f258-f099-4434-bac3-e3f9f7a3bfdf\") " Nov 02 20:15:16.981237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.981004 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-maas-api-service-ca" pod "5a68f258-f099-4434-bac3-e3f9f7a3bfdf" (UID: "5a68f258-f099-4434-bac3-e3f9f7a3bfdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:16.981237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.981196 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:16.982971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:16.982944 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-kube-api-access-zffk6" pod "5a68f258-f099-4434-bac3-e3f9f7a3bfdf" (UID: "5a68f258-f099-4434-bac3-e3f9f7a3bfdf"). InnerVolumeSpecName "kube-api-access-zffk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:17.081777 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.081748 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-kube-api-access-q7t5j\" (UniqueName: \"kubernetes.io/projected/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-kube-api-access-q7t5j\") pod \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " Nov 02 20:15:17.081979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.081965 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-maas-api-service-ca\") pod \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\" (UID: \"f74b7ad1-f775-4370-bb1a-4d71643eaf6f\") " Nov 02 20:15:17.082220 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.082201 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zffk6\" (UniqueName: \"kubernetes.io/projected/5a68f258-f099-4434-bac3-e3f9f7a3bfdf-kube-api-access-zffk6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.082398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.082364 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-maas-api-service-ca" pod "f74b7ad1-f775-4370-bb1a-4d71643eaf6f" (UID: "f74b7ad1-f775-4370-bb1a-4d71643eaf6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:17.083926 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.083896 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-kube-api-access-q7t5j" pod "f74b7ad1-f775-4370-bb1a-4d71643eaf6f" (UID: "f74b7ad1-f775-4370-bb1a-4d71643eaf6f"). InnerVolumeSpecName "kube-api-access-q7t5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:17.183535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.183498 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.183535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.183535 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7t5j\" (UniqueName: \"kubernetes.io/projected/f74b7ad1-f775-4370-bb1a-4d71643eaf6f-kube-api-access-q7t5j\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.607842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.607811 4966 generic.go:356] "Generic (PLEG): container finished" podID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerID="ce1fd17b32cefb3c86d2e362cac4402de2dc075414719d4db30ed304a31385dd" exitCode=60 Nov 02 20:15:17.608063 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.607896 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerDied","Data":"ce1fd17b32cefb3c86d2e362cac4402de2dc075414719d4db30ed304a31385dd"} Nov 02 20:15:17.608063 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.607958 4966 scope.go:122] "RemoveContainer" containerID="be6a6c64ab508eb7a98bbe51e86d9449101206df5179d99f7e93e03ea38c8e02" Nov 02 20:15:17.608383 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.608349 4966 scope.go:122] "RemoveContainer" containerID="ce1fd17b32cefb3c86d2e362cac4402de2dc075414719d4db30ed304a31385dd" Nov 02 20:15:17.608718 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:17.608691 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg_odh-ai-gateway-infra(df678d54-f0bd-4b95-b86a-2c67ae313fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" Nov 02 20:15:17.609534 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.609515 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" Nov 02 20:15:17.609618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.609541 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm" event={"ID":"f74b7ad1-f775-4370-bb1a-4d71643eaf6f","Type":"ContainerDied","Data":"982e70940d4e44978a86b922f9f0f0623c111ec6004ea59a5711d7d1beaac483"} Nov 02 20:15:17.615115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.615091 4966 generic.go:356] "Generic (PLEG): container finished" podID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerID="00c550698d3aafcf51f7301713101f308123782941a2b0dbdbc5747fe7bedbd0" exitCode=60 Nov 02 20:15:17.615234 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.615164 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerDied","Data":"00c550698d3aafcf51f7301713101f308123782941a2b0dbdbc5747fe7bedbd0"} Nov 02 20:15:17.615584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.615564 4966 scope.go:122] "RemoveContainer" containerID="00c550698d3aafcf51f7301713101f308123782941a2b0dbdbc5747fe7bedbd0" Nov 02 20:15:17.615861 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:17.615834 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv_odh-ai-gateway-infra(4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" Nov 02 20:15:17.616656 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.616632 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" Nov 02 20:15:17.616757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.616654 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr" event={"ID":"5a68f258-f099-4434-bac3-e3f9f7a3bfdf","Type":"ContainerDied","Data":"159209fdec2d00de3c84367c9bfb7b30f3a08d0880667b454b18e0dcebd4eef3"} Nov 02 20:15:17.625022 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.625004 4966 scope.go:122] "RemoveContainer" containerID="7333f6a0491b6898a8f7d93d057b931afab2bef92b63a1a6973e96c5934baf5b" Nov 02 20:15:17.634656 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.634635 4966 scope.go:122] "RemoveContainer" containerID="d06f031753257b508718cfd53654ac32d8ccd1d3854e04698aaa813d6b13be99" Nov 02 20:15:17.650341 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.650281 4966 scope.go:122] "RemoveContainer" containerID="1ea567fc119771badc0fc6f8c4224056d8fb589bcf4b4a480ad9d96cdc66be40" Nov 02 20:15:17.701146 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.701112 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg"] Nov 02 20:15:17.810679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.810640 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w"] Nov 02 20:15:17.824525 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.824494 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm"] Nov 02 20:15:17.828072 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.828044 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-24ef8b-29894175-szpbm"] Nov 02 20:15:17.913286 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.913245 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n"] Nov 02 20:15:17.926852 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.926822 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr"] Nov 02 20:15:17.934357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.934323 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-28fa24-29894175-thtkr"] Nov 02 20:15:17.968380 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.967593 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:17.992877 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.992847 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5addd05e-f0bb-40aa-a86c-dce2c22129fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5addd05e-f0bb-40aa-a86c-dce2c22129fb-maas-api-service-ca\") pod \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " Nov 02 20:15:17.993033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.992891 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5addd05e-f0bb-40aa-a86c-dce2c22129fb-kube-api-access-jztn6\" (UniqueName: \"kubernetes.io/projected/5addd05e-f0bb-40aa-a86c-dce2c22129fb-kube-api-access-jztn6\") pod \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\" (UID: \"5addd05e-f0bb-40aa-a86c-dce2c22129fb\") " Nov 02 20:15:17.993679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.993617 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5addd05e-f0bb-40aa-a86c-dce2c22129fb-maas-api-service-ca" pod "5addd05e-f0bb-40aa-a86c-dce2c22129fb" (UID: "5addd05e-f0bb-40aa-a86c-dce2c22129fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:17.996241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:17.996206 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5addd05e-f0bb-40aa-a86c-dce2c22129fb-kube-api-access-jztn6" pod "5addd05e-f0bb-40aa-a86c-dce2c22129fb" (UID: "5addd05e-f0bb-40aa-a86c-dce2c22129fb"). InnerVolumeSpecName "kube-api-access-jztn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.022220 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.022184 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl"] Nov 02 20:15:18.068689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.068659 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:18.094113 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.094074 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5addd05e-f0bb-40aa-a86c-dce2c22129fb-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.094113 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.094112 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jztn6\" (UniqueName: \"kubernetes.io/projected/5addd05e-f0bb-40aa-a86c-dce2c22129fb-kube-api-access-jztn6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.189395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.187612 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" path="/var/lib/kubelet/pods/5a68f258-f099-4434-bac3-e3f9f7a3bfdf/volumes" Nov 02 20:15:18.189395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.188350 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" path="/var/lib/kubelet/pods/f74b7ad1-f775-4370-bb1a-4d71643eaf6f/volumes" Nov 02 20:15:18.192524 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.192498 4966 scope.go:122] "RemoveContainer" containerID="e262bc6c5d4a48c5547b3d3f9d165861857e77217ac97e8c567a928d0f9d316e" Nov 02 20:15:18.193468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.193454 4966 scope.go:122] "RemoveContainer" containerID="82bc5aacc2a5ca469893b2ef007a4643641e60fbc7ca63b7a76308c1f9a60884" Nov 02 20:15:18.194291 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.193887 4966 scope.go:122] "RemoveContainer" containerID="4507d83f4eb850cb6c9c944e2527cbf720b5115521366553474d1cbb60f56b37" Nov 02 20:15:18.196073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.195975 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04eecc2-cc32-4536-8a4b-de0b553306c1-kube-api-access-kvztz\" (UniqueName: \"kubernetes.io/projected/d04eecc2-cc32-4536-8a4b-de0b553306c1-kube-api-access-kvztz\") pod \"d04eecc2-cc32-4536-8a4b-de0b553306c1\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " Nov 02 20:15:18.196139 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.196122 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04eecc2-cc32-4536-8a4b-de0b553306c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04eecc2-cc32-4536-8a4b-de0b553306c1-maas-api-service-ca\") pod \"d04eecc2-cc32-4536-8a4b-de0b553306c1\" (UID: \"d04eecc2-cc32-4536-8a4b-de0b553306c1\") " Nov 02 20:15:18.197057 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.197033 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04eecc2-cc32-4536-8a4b-de0b553306c1-maas-api-service-ca" pod "d04eecc2-cc32-4536-8a4b-de0b553306c1" (UID: "d04eecc2-cc32-4536-8a4b-de0b553306c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.200677 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.199111 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04eecc2-cc32-4536-8a4b-de0b553306c1-kube-api-access-kvztz" pod "d04eecc2-cc32-4536-8a4b-de0b553306c1" (UID: "d04eecc2-cc32-4536-8a4b-de0b553306c1"). InnerVolumeSpecName "kube-api-access-kvztz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.233230 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.233206 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv"] Nov 02 20:15:18.257321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.257287 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:18.298266 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.298243 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04eecc2-cc32-4536-8a4b-de0b553306c1-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.298266 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.298267 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvztz\" (UniqueName: \"kubernetes.io/projected/d04eecc2-cc32-4536-8a4b-de0b553306c1-kube-api-access-kvztz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.398809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.398774 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331eb841-ecb1-4fca-813e-842e3e9f0529-kube-api-access-szqmb\" (UniqueName: \"kubernetes.io/projected/331eb841-ecb1-4fca-813e-842e3e9f0529-kube-api-access-szqmb\") pod \"331eb841-ecb1-4fca-813e-842e3e9f0529\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " Nov 02 20:15:18.399026 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.398885 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331eb841-ecb1-4fca-813e-842e3e9f0529-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331eb841-ecb1-4fca-813e-842e3e9f0529-maas-api-service-ca\") pod \"331eb841-ecb1-4fca-813e-842e3e9f0529\" (UID: \"331eb841-ecb1-4fca-813e-842e3e9f0529\") " Nov 02 20:15:18.399276 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.399246 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331eb841-ecb1-4fca-813e-842e3e9f0529-maas-api-service-ca" pod "331eb841-ecb1-4fca-813e-842e3e9f0529" (UID: "331eb841-ecb1-4fca-813e-842e3e9f0529"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.399416 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.399391 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331eb841-ecb1-4fca-813e-842e3e9f0529-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.440897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.440813 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331eb841-ecb1-4fca-813e-842e3e9f0529-kube-api-access-szqmb" pod "331eb841-ecb1-4fca-813e-842e3e9f0529" (UID: "331eb841-ecb1-4fca-813e-842e3e9f0529"). InnerVolumeSpecName "kube-api-access-szqmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.500526 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.500487 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szqmb\" (UniqueName: \"kubernetes.io/projected/331eb841-ecb1-4fca-813e-842e3e9f0529-kube-api-access-szqmb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.628830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.628800 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" Nov 02 20:15:18.628995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.628822 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl" event={"ID":"331eb841-ecb1-4fca-813e-842e3e9f0529","Type":"ContainerDied","Data":"9d11ce58fe2980c9e99193fc83179298ccdc96fbdec5121dfcbadc185f67a2c4"} Nov 02 20:15:18.628995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.628891 4966 scope.go:122] "RemoveContainer" containerID="0af4d6aec581de3f1c79800f3ab16b8008eba7d18a0a07837a6d3b97b5a0333f" Nov 02 20:15:18.634053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.634022 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" event={"ID":"d04eecc2-cc32-4536-8a4b-de0b553306c1","Type":"ContainerDied","Data":"98827974d10a501546b0cd91450647bf9c8b5661fb6886a1c463d9926f37d7b5"} Nov 02 20:15:18.634185 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.634049 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n" Nov 02 20:15:18.636066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.636041 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" event={"ID":"5addd05e-f0bb-40aa-a86c-dce2c22129fb","Type":"ContainerDied","Data":"fed2053a6bec52e1657858646640c9c9237cb9b99191e3e404587a49e26eb2ce"} Nov 02 20:15:18.636188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.636091 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w" Nov 02 20:15:18.643651 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.643631 4966 scope.go:122] "RemoveContainer" containerID="fd7d00716f6b02f795a23f025a29e471b4b4bd54da3e177539c9d06a463d616a" Nov 02 20:15:18.660818 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.660799 4966 scope.go:122] "RemoveContainer" containerID="40188ca4cca3c10e2a3c8024390cb275eca021c5a7a67b7ba683dcc3389237d9" Nov 02 20:15:18.835218 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.835181 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w"] Nov 02 20:15:18.842308 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.842254 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-838bc1-29894175-nzl8w"] Nov 02 20:15:18.898675 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.898642 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:18.905117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.905084 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-kube-api-access-r5dzz\" (UniqueName: \"kubernetes.io/projected/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-kube-api-access-r5dzz\") pod \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " Nov 02 20:15:18.905261 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.905149 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-maas-api-service-ca\") pod \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\" (UID: \"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0\") " Nov 02 20:15:18.905577 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.905550 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-maas-api-service-ca" pod "4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" (UID: "4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.905716 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.905695 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.907500 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.907476 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-kube-api-access-r5dzz" pod "4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" (UID: "4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0"). InnerVolumeSpecName "kube-api-access-r5dzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.937910 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.937818 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:18.944063 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.944033 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n"] Nov 02 20:15:18.949359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:18.949330 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-13e739-29894175-bg47n"] Nov 02 20:15:19.006468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.006363 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df678d54-f0bd-4b95-b86a-2c67ae313fcd-kube-api-access-sjd27\" (UniqueName: \"kubernetes.io/projected/df678d54-f0bd-4b95-b86a-2c67ae313fcd-kube-api-access-sjd27\") pod \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " Nov 02 20:15:19.006468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.006443 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df678d54-f0bd-4b95-b86a-2c67ae313fcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df678d54-f0bd-4b95-b86a-2c67ae313fcd-maas-api-service-ca\") pod \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\" (UID: \"df678d54-f0bd-4b95-b86a-2c67ae313fcd\") " Nov 02 20:15:19.006873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.006848 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dzz\" (UniqueName: \"kubernetes.io/projected/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0-kube-api-access-r5dzz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.006959 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.006861 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df678d54-f0bd-4b95-b86a-2c67ae313fcd-maas-api-service-ca" pod "df678d54-f0bd-4b95-b86a-2c67ae313fcd" (UID: "df678d54-f0bd-4b95-b86a-2c67ae313fcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:19.008837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.008814 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df678d54-f0bd-4b95-b86a-2c67ae313fcd-kube-api-access-sjd27" pod "df678d54-f0bd-4b95-b86a-2c67ae313fcd" (UID: "df678d54-f0bd-4b95-b86a-2c67ae313fcd"). InnerVolumeSpecName "kube-api-access-sjd27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:19.046618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.046581 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl"] Nov 02 20:15:19.053954 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.053922 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-4848d2-29894175-l9jtl"] Nov 02 20:15:19.108752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.108709 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjd27\" (UniqueName: \"kubernetes.io/projected/df678d54-f0bd-4b95-b86a-2c67ae313fcd-kube-api-access-sjd27\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.108752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.108753 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df678d54-f0bd-4b95-b86a-2c67ae313fcd-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.644604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.644508 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" Nov 02 20:15:19.644604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.644521 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv" event={"ID":"4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0","Type":"ContainerDied","Data":"36ecf0105b9dae3229ddba13d3c36985132b363a42714838468a7323fe34e422"} Nov 02 20:15:19.645156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.644676 4966 scope.go:122] "RemoveContainer" containerID="00c550698d3aafcf51f7301713101f308123782941a2b0dbdbc5747fe7bedbd0" Nov 02 20:15:19.646743 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.646719 4966 generic.go:356] "Generic (PLEG): container finished" podID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerID="669b3b53d0138d48f5f77eaf15cd2573cd53de152f40be8aa9f70cda952dc1f5" exitCode=60 Nov 02 20:15:19.646885 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.646827 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" event={"ID":"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba","Type":"ContainerDied","Data":"669b3b53d0138d48f5f77eaf15cd2573cd53de152f40be8aa9f70cda952dc1f5"} Nov 02 20:15:19.647335 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.647281 4966 scope.go:122] "RemoveContainer" containerID="669b3b53d0138d48f5f77eaf15cd2573cd53de152f40be8aa9f70cda952dc1f5" Nov 02 20:15:19.647669 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:19.647638 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh_odh-ai-gateway-infra(6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" Nov 02 20:15:19.648453 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.648428 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" Nov 02 20:15:19.648539 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.648450 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg" event={"ID":"df678d54-f0bd-4b95-b86a-2c67ae313fcd","Type":"ContainerDied","Data":"d3d2908a44aa0495b35dad5010fd5503975c7be79e7764ada42434583370406d"} Nov 02 20:15:19.650697 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.650669 4966 generic.go:356] "Generic (PLEG): container finished" podID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerID="86695685d444c682041f2e93c4e83734a8bb3945fc8515e802fab1fe1a20fe08" exitCode=60 Nov 02 20:15:19.650778 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.650737 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" event={"ID":"d3a3287b-f705-4b73-aed1-5c04e4febbc2","Type":"ContainerDied","Data":"86695685d444c682041f2e93c4e83734a8bb3945fc8515e802fab1fe1a20fe08"} Nov 02 20:15:19.651171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.651152 4966 scope.go:122] "RemoveContainer" containerID="86695685d444c682041f2e93c4e83734a8bb3945fc8515e802fab1fe1a20fe08" Nov 02 20:15:19.651490 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:19.651468 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr_odh-ai-gateway-infra(d3a3287b-f705-4b73-aed1-5c04e4febbc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" Nov 02 20:15:19.653014 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.652989 4966 generic.go:356] "Generic (PLEG): container finished" podID="a825edf8-6b57-40d8-81a4-873300069d62" containerID="d1aa4dcd97375ff80a3ea08bafb3f057db5e23fbb90ab742049966fcee5ec16f" exitCode=60 Nov 02 20:15:19.653130 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.653016 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" event={"ID":"a825edf8-6b57-40d8-81a4-873300069d62","Type":"ContainerDied","Data":"d1aa4dcd97375ff80a3ea08bafb3f057db5e23fbb90ab742049966fcee5ec16f"} Nov 02 20:15:19.653434 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.653411 4966 scope.go:122] "RemoveContainer" containerID="d1aa4dcd97375ff80a3ea08bafb3f057db5e23fbb90ab742049966fcee5ec16f" Nov 02 20:15:19.653749 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:19.653720 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc_odh-ai-gateway-infra(a825edf8-6b57-40d8-81a4-873300069d62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" podUID="a825edf8-6b57-40d8-81a4-873300069d62" Nov 02 20:15:19.657198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.657157 4966 scope.go:122] "RemoveContainer" containerID="e262bc6c5d4a48c5547b3d3f9d165861857e77217ac97e8c567a928d0f9d316e" Nov 02 20:15:19.671237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.671212 4966 scope.go:122] "RemoveContainer" containerID="ce1fd17b32cefb3c86d2e362cac4402de2dc075414719d4db30ed304a31385dd" Nov 02 20:15:19.681111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.681090 4966 scope.go:122] "RemoveContainer" containerID="4507d83f4eb850cb6c9c944e2527cbf720b5115521366553474d1cbb60f56b37" Nov 02 20:15:19.695101 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:19.695075 4966 scope.go:122] "RemoveContainer" containerID="82bc5aacc2a5ca469893b2ef007a4643641e60fbc7ca63b7a76308c1f9a60884" Nov 02 20:15:20.008389 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.008344 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv"] Nov 02 20:15:20.017861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.017828 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-19d134-29894175-4m7xv"] Nov 02 20:15:20.118276 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.118239 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg"] Nov 02 20:15:20.124579 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.124546 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7025fa-29894175-b79mg"] Nov 02 20:15:20.193245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.193209 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" path="/var/lib/kubelet/pods/331eb841-ecb1-4fca-813e-842e3e9f0529/volumes" Nov 02 20:15:20.193669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.193652 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" path="/var/lib/kubelet/pods/4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0/volumes" Nov 02 20:15:20.194040 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.194026 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" path="/var/lib/kubelet/pods/5addd05e-f0bb-40aa-a86c-dce2c22129fb/volumes" Nov 02 20:15:20.194751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.194735 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" path="/var/lib/kubelet/pods/d04eecc2-cc32-4536-8a4b-de0b553306c1/volumes" Nov 02 20:15:20.195111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.195099 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" path="/var/lib/kubelet/pods/df678d54-f0bd-4b95-b86a-2c67ae313fcd/volumes" Nov 02 20:15:20.502117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502079 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr"] Nov 02 20:15:20.502660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502643 4966 cpu_manager.go:414] "containerMap: removing container" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.502660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502660 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502673 4966 cpu_manager.go:414] "containerMap: removing container" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502678 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502689 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e5bf48b0-4073-420d-83b0-4f577342622e" containerName="main" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502695 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bf48b0-4073-420d-83b0-4f577342622e" containerName="main" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502701 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502706 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502714 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502719 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502726 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502730 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502736 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502741 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502752 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502756 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502763 4966 cpu_manager.go:414] "containerMap: removing container" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502768 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502775 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502779 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502787 4966 cpu_manager.go:414] "containerMap: removing container" podUID="b4d190c1-32e8-4b22-9b4e-0eedcba467d9" containerName="revoke-keys" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502792 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d190c1-32e8-4b22-9b4e-0eedcba467d9" containerName="revoke-keys" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502800 4966 cpu_manager.go:414] "containerMap: removing container" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502804 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:20.502803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502811 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502816 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502823 4966 cpu_manager.go:414] "containerMap: removing container" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502828 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502841 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502846 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502853 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502858 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502866 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502871 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502882 4966 cpu_manager.go:414] "containerMap: removing container" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502886 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502894 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502907 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502916 4966 cpu_manager.go:414] "containerMap: removing container" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.502921 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503005 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503013 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503019 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503026 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d190c1-32e8-4b22-9b4e-0eedcba467d9" containerName="revoke-keys" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503032 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503037 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503043 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503049 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503054 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503058 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503064 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503069 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503074 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a68f258-f099-4434-bac3-e3f9f7a3bfdf" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503080 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="971c7ec4-2c35-44c3-afdb-6e2e9df1a7b1" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503086 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503092 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503098 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="df678d54-f0bd-4b95-b86a-2c67ae313fcd" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503104 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bf48b0-4073-420d-83b0-4f577342622e" containerName="main" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503109 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:20.503564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.503114 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:20.508159 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.508137 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.524669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.524642 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9nn\" (UniqueName: \"kubernetes.io/projected/a796d742-fd83-43aa-9cae-a34052c4d20c-kube-api-access-jm9nn\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.524809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.524705 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.524873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.524851 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.556650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.556599 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:15:20.559996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.559963 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr"] Nov 02 20:15:20.625971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.625939 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.626165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.626050 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.626165 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.626087 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9nn\" (UniqueName: \"kubernetes.io/projected/a796d742-fd83-43aa-9cae-a34052c4d20c-kube-api-access-jm9nn\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.626680 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.626654 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.628560 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.628532 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.679742 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.679709 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9nn\" (UniqueName: \"kubernetes.io/projected/a796d742-fd83-43aa-9cae-a34052c4d20c-kube-api-access-jm9nn\") pod \"maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.810851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.810755 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc"] Nov 02 20:15:20.821616 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.821576 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:20.925642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.925604 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh"] Nov 02 20:15:20.970167 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:20.970131 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:21.030784 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.030748 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a825edf8-6b57-40d8-81a4-873300069d62-kube-api-access-6ksk8\" (UniqueName: \"kubernetes.io/projected/a825edf8-6b57-40d8-81a4-873300069d62-kube-api-access-6ksk8\") pod \"a825edf8-6b57-40d8-81a4-873300069d62\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " Nov 02 20:15:21.030960 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.030859 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a825edf8-6b57-40d8-81a4-873300069d62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a825edf8-6b57-40d8-81a4-873300069d62-maas-api-service-ca\") pod \"a825edf8-6b57-40d8-81a4-873300069d62\" (UID: \"a825edf8-6b57-40d8-81a4-873300069d62\") " Nov 02 20:15:21.031588 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.031558 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a825edf8-6b57-40d8-81a4-873300069d62-maas-api-service-ca" pod "a825edf8-6b57-40d8-81a4-873300069d62" (UID: "a825edf8-6b57-40d8-81a4-873300069d62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:21.034823 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.034792 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a825edf8-6b57-40d8-81a4-873300069d62-kube-api-access-6ksk8" pod "a825edf8-6b57-40d8-81a4-873300069d62" (UID: "a825edf8-6b57-40d8-81a4-873300069d62"). InnerVolumeSpecName "kube-api-access-6ksk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:21.099582 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.099555 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:21.132526 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.132494 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-kube-api-access-dwsdw\" (UniqueName: \"kubernetes.io/projected/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-kube-api-access-dwsdw\") pod \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " Nov 02 20:15:21.132703 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.132660 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-maas-api-service-ca\") pod \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\" (UID: \"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba\") " Nov 02 20:15:21.132948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.132930 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ksk8\" (UniqueName: \"kubernetes.io/projected/a825edf8-6b57-40d8-81a4-873300069d62-kube-api-access-6ksk8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.132948 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.132947 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a825edf8-6b57-40d8-81a4-873300069d62-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.133097 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.132993 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-maas-api-service-ca" pod "6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" (UID: "6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:21.134600 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.134576 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-kube-api-access-dwsdw" pod "6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" (UID: "6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba"). InnerVolumeSpecName "kube-api-access-dwsdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:21.205390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.205356 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr"] Nov 02 20:15:21.206128 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:21.206101 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda796d742_fd83_43aa_9cae_a34052c4d20c.slice/crio-1007c69c99f4c9a1491f0f43d363f7b99e931cf2d8ace5e9f42d7e10aff711a2 WatchSource:0}: Error finding container 1007c69c99f4c9a1491f0f43d363f7b99e931cf2d8ace5e9f42d7e10aff711a2: Status 404 returned error can't find the container with id 1007c69c99f4c9a1491f0f43d363f7b99e931cf2d8ace5e9f42d7e10aff711a2 Nov 02 20:15:21.234439 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.234411 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.234439 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.234437 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwsdw\" (UniqueName: \"kubernetes.io/projected/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba-kube-api-access-dwsdw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.234743 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.234717 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr"] Nov 02 20:15:21.405433 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.405396 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:21.436910 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.436881 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a3287b-f705-4b73-aed1-5c04e4febbc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a3287b-f705-4b73-aed1-5c04e4febbc2-maas-api-service-ca\") pod \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " Nov 02 20:15:21.437065 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.436964 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a3287b-f705-4b73-aed1-5c04e4febbc2-kube-api-access-249dw\" (UniqueName: \"kubernetes.io/projected/d3a3287b-f705-4b73-aed1-5c04e4febbc2-kube-api-access-249dw\") pod \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\" (UID: \"d3a3287b-f705-4b73-aed1-5c04e4febbc2\") " Nov 02 20:15:21.437312 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.437273 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a3287b-f705-4b73-aed1-5c04e4febbc2-maas-api-service-ca" pod "d3a3287b-f705-4b73-aed1-5c04e4febbc2" (UID: "d3a3287b-f705-4b73-aed1-5c04e4febbc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:21.437418 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.437400 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a3287b-f705-4b73-aed1-5c04e4febbc2-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.439701 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.439669 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a3287b-f705-4b73-aed1-5c04e4febbc2-kube-api-access-249dw" pod "d3a3287b-f705-4b73-aed1-5c04e4febbc2" (UID: "d3a3287b-f705-4b73-aed1-5c04e4febbc2"). InnerVolumeSpecName "kube-api-access-249dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:21.538906 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.538871 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-249dw\" (UniqueName: \"kubernetes.io/projected/d3a3287b-f705-4b73-aed1-5c04e4febbc2-kube-api-access-249dw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:21.675999 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.675963 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" Nov 02 20:15:21.675999 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.675989 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr" event={"ID":"d3a3287b-f705-4b73-aed1-5c04e4febbc2","Type":"ContainerDied","Data":"97d648d28a21190bd00587a8422eb6ad6bf2df89dee15c29189351ad24289eef"} Nov 02 20:15:21.676235 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.676060 4966 scope.go:122] "RemoveContainer" containerID="86695685d444c682041f2e93c4e83734a8bb3945fc8515e802fab1fe1a20fe08" Nov 02 20:15:21.677752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.677733 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" Nov 02 20:15:21.677902 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.677765 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc" event={"ID":"a825edf8-6b57-40d8-81a4-873300069d62","Type":"ContainerDied","Data":"04405ebc980ebf0254f0d00c34c4437e0a7d0ac496a3694b79353a353764d39d"} Nov 02 20:15:21.679618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.679581 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" event={"ID":"6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba","Type":"ContainerDied","Data":"db3618b89483f787342f200e4c0c2140b2056ae0db66a3d66f4b4eb25ae90247"} Nov 02 20:15:21.679761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.679606 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh" Nov 02 20:15:21.682564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.682537 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" event={"ID":"a796d742-fd83-43aa-9cae-a34052c4d20c","Type":"ContainerStarted","Data":"e86d9150b665b4d2f889093754b394960cd1f6848f330c33c92af2d7fbecae07"} Nov 02 20:15:21.682720 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.682587 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" event={"ID":"a796d742-fd83-43aa-9cae-a34052c4d20c","Type":"ContainerStarted","Data":"1007c69c99f4c9a1491f0f43d363f7b99e931cf2d8ace5e9f42d7e10aff711a2"} Nov 02 20:15:21.691367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.690126 4966 scope.go:122] "RemoveContainer" containerID="d1aa4dcd97375ff80a3ea08bafb3f057db5e23fbb90ab742049966fcee5ec16f" Nov 02 20:15:21.702864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.702842 4966 scope.go:122] "RemoveContainer" containerID="669b3b53d0138d48f5f77eaf15cd2573cd53de152f40be8aa9f70cda952dc1f5" Nov 02 20:15:21.833205 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.833165 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr"] Nov 02 20:15:21.837129 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.837096 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-282875-29894175-6r9gr"] Nov 02 20:15:21.943923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:21.943815 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" podStartSLOduration=1.943795629 podStartE2EDuration="1.943795629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:21.93842484 +0000 UTC m=+2827.887080692" watchObservedRunningTime="2026-11-02 20:15:21.943795629 +0000 UTC m=+2827.892451469" Nov 02 20:15:22.044291 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.044251 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc"] Nov 02 20:15:22.049081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.049042 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-d1aaed-29894175-ntglc"] Nov 02 20:15:22.150647 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.150611 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh"] Nov 02 20:15:22.156402 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.155905 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26cfbe-29894175-f9crh"] Nov 02 20:15:22.186409 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.185863 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" path="/var/lib/kubelet/pods/6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba/volumes" Nov 02 20:15:22.195333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.195217 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a825edf8-6b57-40d8-81a4-873300069d62" path="/var/lib/kubelet/pods/a825edf8-6b57-40d8-81a4-873300069d62/volumes" Nov 02 20:15:22.196175 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.196146 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" path="/var/lib/kubelet/pods/d3a3287b-f705-4b73-aed1-5c04e4febbc2/volumes" Nov 02 20:15:22.694294 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.694260 4966 generic.go:356] "Generic (PLEG): container finished" podID="a796d742-fd83-43aa-9cae-a34052c4d20c" containerID="e86d9150b665b4d2f889093754b394960cd1f6848f330c33c92af2d7fbecae07" exitCode=0 Nov 02 20:15:22.694759 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:22.694351 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" event={"ID":"a796d742-fd83-43aa-9cae-a34052c4d20c","Type":"ContainerDied","Data":"e86d9150b665b4d2f889093754b394960cd1f6848f330c33c92af2d7fbecae07"} Nov 02 20:15:23.852148 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:23.852122 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:23.965197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:23.965109 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-xks-ca\") pod \"a796d742-fd83-43aa-9cae-a34052c4d20c\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " Nov 02 20:15:23.965197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:23.965184 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-openshift-ca\") pod \"a796d742-fd83-43aa-9cae-a34052c4d20c\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " Nov 02 20:15:23.965457 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:23.965236 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a796d742-fd83-43aa-9cae-a34052c4d20c-kube-api-access-jm9nn\" (UniqueName: \"kubernetes.io/projected/a796d742-fd83-43aa-9cae-a34052c4d20c-kube-api-access-jm9nn\") pod \"a796d742-fd83-43aa-9cae-a34052c4d20c\" (UID: \"a796d742-fd83-43aa-9cae-a34052c4d20c\") " Nov 02 20:15:23.965636 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:23.965605 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-openshift-ca" pod "a796d742-fd83-43aa-9cae-a34052c4d20c" (UID: "a796d742-fd83-43aa-9cae-a34052c4d20c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:23.967500 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:23.967472 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a796d742-fd83-43aa-9cae-a34052c4d20c-kube-api-access-jm9nn" pod "a796d742-fd83-43aa-9cae-a34052c4d20c" (UID: "a796d742-fd83-43aa-9cae-a34052c4d20c"). InnerVolumeSpecName "kube-api-access-jm9nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:23.967591 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:23.967474 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-xks-ca" pod "a796d742-fd83-43aa-9cae-a34052c4d20c" (UID: "a796d742-fd83-43aa-9cae-a34052c4d20c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:24.066438 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:24.066406 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:24.066438 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:24.066435 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a796d742-fd83-43aa-9cae-a34052c4d20c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:24.066438 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:24.066447 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm9nn\" (UniqueName: \"kubernetes.io/projected/a796d742-fd83-43aa-9cae-a34052c4d20c-kube-api-access-jm9nn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:24.710492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:24.710404 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" event={"ID":"a796d742-fd83-43aa-9cae-a34052c4d20c","Type":"ContainerDied","Data":"1007c69c99f4c9a1491f0f43d363f7b99e931cf2d8ace5e9f42d7e10aff711a2"} Nov 02 20:15:24.710492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:24.710457 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr" Nov 02 20:15:24.710492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:24.710469 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1007c69c99f4c9a1491f0f43d363f7b99e931cf2d8ace5e9f42d7e10aff711a2" Nov 02 20:15:25.033434 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:25.033400 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f"] Nov 02 20:15:25.038273 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:25.038237 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc"] Nov 02 20:15:32.232769 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:32.232733 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8"] Nov 02 20:15:32.277089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:32.277056 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-838bc1-cfsv8"] Nov 02 20:15:33.547344 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:33.547279 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5"] Nov 02 20:15:33.593358 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:33.593322 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7025fa-g99g5"] Nov 02 20:15:34.183512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:34.183470 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28cab378-908f-4877-af24-685c7d93e7ae" path="/var/lib/kubelet/pods/28cab378-908f-4877-af24-685c7d93e7ae/volumes" Nov 02 20:15:34.183923 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:34.183907 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e332bca4-b357-4d7d-ba52-6563ce716794" path="/var/lib/kubelet/pods/e332bca4-b357-4d7d-ba52-6563ce716794/volumes" Nov 02 20:15:39.634242 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.634193 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq"] Nov 02 20:15:39.635947 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.635913 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:39.636130 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636116 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:39.636246 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636235 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a796d742-fd83-43aa-9cae-a34052c4d20c" containerName="revoke-keys" Nov 02 20:15:39.636357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636344 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a796d742-fd83-43aa-9cae-a34052c4d20c" containerName="revoke-keys" Nov 02 20:15:39.636476 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636466 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.636554 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636542 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.636634 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636625 4966 cpu_manager.go:414] "containerMap: removing container" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:39.636705 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636695 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:39.636847 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636818 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.636955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.636941 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.637103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637092 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:39.637205 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637194 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:39.637348 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637336 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.637438 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637427 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.637514 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637506 4966 cpu_manager.go:414] "containerMap: removing container" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:39.637583 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637573 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:39.637657 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637649 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:39.637733 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637724 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:39.637834 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637825 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.637909 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.637899 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.644815 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.644529 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:39.645963 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.645939 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:39.646094 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646081 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.646187 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646170 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.646346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646212 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:39.646346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646222 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:39.646346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646239 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.646346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646247 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.646647 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646630 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.646716 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646654 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74b7ad1-f775-4370-bb1a-4d71643eaf6f" containerName="cleanup" Nov 02 20:15:39.646716 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646678 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5addd05e-f0bb-40aa-a86c-dce2c22129fb" containerName="cleanup" Nov 02 20:15:39.646716 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646697 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:39.646716 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646708 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646726 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="331eb841-ecb1-4fca-813e-842e3e9f0529" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646744 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfa89f3-ad5d-42d4-9ab8-5f3f11c167a0" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646755 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646765 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646779 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a3287b-f705-4b73-aed1-5c04e4febbc2" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646790 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04eecc2-cc32-4536-8a4b-de0b553306c1" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646807 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:39.646930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.646824 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a796d742-fd83-43aa-9cae-a34052c4d20c" containerName="revoke-keys" Nov 02 20:15:39.653257 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.653215 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.688771 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.688732 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq"] Nov 02 20:15:39.730947 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.730909 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:15:39.731142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.730954 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:15:39.731142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.731025 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:15:39.747562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.747529 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:15:39.747746 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.747647 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldk9p\" (UniqueName: \"kubernetes.io/projected/71b5ae16-6416-4ca5-8379-ef275bb98b4e-kube-api-access-ldk9p\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.747746 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.747707 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.747746 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.747732 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.849113 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.849074 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldk9p\" (UniqueName: \"kubernetes.io/projected/71b5ae16-6416-4ca5-8379-ef275bb98b4e-kube-api-access-ldk9p\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.849352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.849135 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.849352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.849157 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.849835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.849805 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.851629 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.851603 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.902579 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.902504 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldk9p\" (UniqueName: \"kubernetes.io/projected/71b5ae16-6416-4ca5-8379-ef275bb98b4e-kube-api-access-ldk9p\") pod \"maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:39.968874 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:39.968830 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:40.057282 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.057211 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" containerID="cri-o://dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd" gracePeriod=45 Nov 02 20:15:40.062708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.062644 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" containerID="cri-o://d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0" gracePeriod=45 Nov 02 20:15:40.140207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.140144 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused" Nov 02 20:15:40.348828 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.348708 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq"] Nov 02 20:15:40.355823 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:40.355774 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b5ae16_6416_4ca5_8379_ef275bb98b4e.slice/crio-5e5606935c221ce0e913534119d0ca55ade25ecb0d2d27e0b651df5b72478a22 WatchSource:0}: Error finding container 5e5606935c221ce0e913534119d0ca55ade25ecb0d2d27e0b651df5b72478a22: Status 404 returned error can't find the container with id 5e5606935c221ce0e913534119d0ca55ade25ecb0d2d27e0b651df5b72478a22 Nov 02 20:15:40.360683 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.360659 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 20:15:40.377395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.377367 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 20:15:40.455811 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.455726 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fe8400-2d35-4658-a087-5a12feb990cc-kube-api-access-wgkfk\" (UniqueName: \"kubernetes.io/projected/04fe8400-2d35-4658-a087-5a12feb990cc-kube-api-access-wgkfk\") pod \"04fe8400-2d35-4658-a087-5a12feb990cc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " Nov 02 20:15:40.456087 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456067 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce95b19-3999-488e-9653-50055cf66632-kube-api-access-mlpr9\" (UniqueName: \"kubernetes.io/projected/dce95b19-3999-488e-9653-50055cf66632-kube-api-access-mlpr9\") pod \"dce95b19-3999-488e-9653-50055cf66632\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " Nov 02 20:15:40.456198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456133 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fe8400-2d35-4658-a087-5a12feb990cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe8400-2d35-4658-a087-5a12feb990cc-tls-certs\") pod \"04fe8400-2d35-4658-a087-5a12feb990cc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " Nov 02 20:15:40.456198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456170 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-dshm\") pod \"dce95b19-3999-488e-9653-50055cf66632\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " Nov 02 20:15:40.456356 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456236 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-tmp-dir\") pod \"04fe8400-2d35-4658-a087-5a12feb990cc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " Nov 02 20:15:40.456356 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456272 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-tmp-dir\") pod \"dce95b19-3999-488e-9653-50055cf66632\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " Nov 02 20:15:40.456356 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456324 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-dshm\") pod \"04fe8400-2d35-4658-a087-5a12feb990cc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " Nov 02 20:15:40.456597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456358 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-model-cache\") pod \"04fe8400-2d35-4658-a087-5a12feb990cc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " Nov 02 20:15:40.456597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456413 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce95b19-3999-488e-9653-50055cf66632-tls-certs\" (UniqueName: \"kubernetes.io/secret/dce95b19-3999-488e-9653-50055cf66632-tls-certs\") pod \"dce95b19-3999-488e-9653-50055cf66632\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " Nov 02 20:15:40.456597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456444 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-home\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-home\") pod \"dce95b19-3999-488e-9653-50055cf66632\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " Nov 02 20:15:40.456597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456477 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-model-cache\") pod \"dce95b19-3999-488e-9653-50055cf66632\" (UID: \"dce95b19-3999-488e-9653-50055cf66632\") " Nov 02 20:15:40.456597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.456517 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-home\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-home\") pod \"04fe8400-2d35-4658-a087-5a12feb990cc\" (UID: \"04fe8400-2d35-4658-a087-5a12feb990cc\") " Nov 02 20:15:40.457934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.457202 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-model-cache" pod "04fe8400-2d35-4658-a087-5a12feb990cc" (UID: "04fe8400-2d35-4658-a087-5a12feb990cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.457934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.457281 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-home" pod "04fe8400-2d35-4658-a087-5a12feb990cc" (UID: "04fe8400-2d35-4658-a087-5a12feb990cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.457934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.457517 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-home" pod "dce95b19-3999-488e-9653-50055cf66632" (UID: "dce95b19-3999-488e-9653-50055cf66632"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.457934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.457554 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-tmp-dir" pod "04fe8400-2d35-4658-a087-5a12feb990cc" (UID: "04fe8400-2d35-4658-a087-5a12feb990cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.457934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.457723 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-model-cache" pod "dce95b19-3999-488e-9653-50055cf66632" (UID: "dce95b19-3999-488e-9653-50055cf66632"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.457934 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.457741 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-tmp-dir" pod "dce95b19-3999-488e-9653-50055cf66632" (UID: "dce95b19-3999-488e-9653-50055cf66632"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.538763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.538715 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-dshm" pod "04fe8400-2d35-4658-a087-5a12feb990cc" (UID: "04fe8400-2d35-4658-a087-5a12feb990cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.538763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.538756 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fe8400-2d35-4658-a087-5a12feb990cc-tls-certs" pod "04fe8400-2d35-4658-a087-5a12feb990cc" (UID: "04fe8400-2d35-4658-a087-5a12feb990cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:40.538986 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.538725 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-dshm" pod "dce95b19-3999-488e-9653-50055cf66632" (UID: "dce95b19-3999-488e-9653-50055cf66632"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:40.538986 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.538759 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe8400-2d35-4658-a087-5a12feb990cc-kube-api-access-wgkfk" pod "04fe8400-2d35-4658-a087-5a12feb990cc" (UID: "04fe8400-2d35-4658-a087-5a12feb990cc"). InnerVolumeSpecName "kube-api-access-wgkfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:40.539086 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.539066 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce95b19-3999-488e-9653-50055cf66632-kube-api-access-mlpr9" pod "dce95b19-3999-488e-9653-50055cf66632" (UID: "dce95b19-3999-488e-9653-50055cf66632"). InnerVolumeSpecName "kube-api-access-mlpr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:40.539148 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.539128 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce95b19-3999-488e-9653-50055cf66632-tls-certs" pod "dce95b19-3999-488e-9653-50055cf66632" (UID: "dce95b19-3999-488e-9653-50055cf66632"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:40.558714 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558686 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlpr9\" (UniqueName: \"kubernetes.io/projected/dce95b19-3999-488e-9653-50055cf66632-kube-api-access-mlpr9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558717 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe8400-2d35-4658-a087-5a12feb990cc-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558733 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558747 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558759 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558772 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558784 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558797 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce95b19-3999-488e-9653-50055cf66632-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558809 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.558836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558826 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce95b19-3999-488e-9653-50055cf66632-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.559150 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558838 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe8400-2d35-4658-a087-5a12feb990cc-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.559150 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.558851 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgkfk\" (UniqueName: \"kubernetes.io/projected/04fe8400-2d35-4658-a087-5a12feb990cc-kube-api-access-wgkfk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:40.836800 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.836763 4966 generic.go:356] "Generic (PLEG): container finished" podID="04fe8400-2d35-4658-a087-5a12feb990cc" containerID="d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0" exitCode=0 Nov 02 20:15:40.836800 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.836807 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" Nov 02 20:15:40.837411 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.836857 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" event={"ID":"04fe8400-2d35-4658-a087-5a12feb990cc","Type":"ContainerDied","Data":"d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0"} Nov 02 20:15:40.837411 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.836903 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc" event={"ID":"04fe8400-2d35-4658-a087-5a12feb990cc","Type":"ContainerDied","Data":"2a57d757ce49d5ddde0fdb27e8dc6a170dc441e469a8d061eec3d575d3bd0ba3"} Nov 02 20:15:40.837411 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.836926 4966 scope.go:122] "RemoveContainer" containerID="d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0" Nov 02 20:15:40.840990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.840954 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" event={"ID":"71b5ae16-6416-4ca5-8379-ef275bb98b4e","Type":"ContainerStarted","Data":"e7d010d367b39f1a0f4bc114fd947defe0bb1e18b184c1267ee7b271e5044a87"} Nov 02 20:15:40.841123 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.841021 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" event={"ID":"71b5ae16-6416-4ca5-8379-ef275bb98b4e","Type":"ContainerStarted","Data":"5e5606935c221ce0e913534119d0ca55ade25ecb0d2d27e0b651df5b72478a22"} Nov 02 20:15:40.842501 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.842472 4966 generic.go:356] "Generic (PLEG): container finished" podID="dce95b19-3999-488e-9653-50055cf66632" containerID="dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd" exitCode=0 Nov 02 20:15:40.842626 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.842512 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" Nov 02 20:15:40.842626 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.842594 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" event={"ID":"dce95b19-3999-488e-9653-50055cf66632","Type":"ContainerDied","Data":"dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd"} Nov 02 20:15:40.842727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.842641 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f" event={"ID":"dce95b19-3999-488e-9653-50055cf66632","Type":"ContainerDied","Data":"626c024a1d856d3003c939af626d89b42003c9eaeb6cfa87d79137b167b929f0"} Nov 02 20:15:40.849745 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.849721 4966 scope.go:122] "RemoveContainer" containerID="d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0" Nov 02 20:15:40.850060 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:40.850038 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0\": container with ID starting with d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0 not found: ID does not exist" containerID="d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0" Nov 02 20:15:40.850146 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.850072 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0"} err="failed to get container status \"d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0\": rpc error: code = NotFound desc = could not find container \"d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0\": container with ID starting with d1d75a282302f73d7b0996319b2271fc91e6fad35287817388a0682fdc243ed0 not found: ID does not exist" Nov 02 20:15:40.850146 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.850097 4966 scope.go:122] "RemoveContainer" containerID="dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd" Nov 02 20:15:40.881645 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.881622 4966 scope.go:122] "RemoveContainer" containerID="dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd" Nov 02 20:15:40.882037 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:40.882016 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd\": container with ID starting with dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd not found: ID does not exist" containerID="dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd" Nov 02 20:15:40.882099 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.882047 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd"} err="failed to get container status \"dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd\": rpc error: code = NotFound desc = could not find container \"dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd\": container with ID starting with dc7625701a189efeda7d9a451344cecacfe83275fd7527d26f9cca65842c48bd not found: ID does not exist" Nov 02 20:15:40.989130 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.989062 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc"] Nov 02 20:15:40.992402 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:40.992371 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/simulator-w3-5c46af-kserve-9db7c84fc-npskc"] Nov 02 20:15:41.093276 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.093200 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f"] Nov 02 20:15:41.096838 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.096809 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz"] Nov 02 20:15:41.097381 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097363 4966 cpu_manager.go:414] "containerMap: removing container" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" Nov 02 20:15:41.097381 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097381 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" Nov 02 20:15:41.097537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097396 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:41.097537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097402 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:41.097537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097429 4966 cpu_manager.go:414] "containerMap: removing container" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" Nov 02 20:15:41.097537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097437 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" Nov 02 20:15:41.097537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097528 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" containerName="main" Nov 02 20:15:41.097537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097537 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc98cf7-9c69-4dbb-ba1a-0ab63d778fba" containerName="cleanup" Nov 02 20:15:41.097820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097544 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a825edf8-6b57-40d8-81a4-873300069d62" containerName="cleanup" Nov 02 20:15:41.097820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.097555 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce95b19-3999-488e-9653-50055cf66632" containerName="main" Nov 02 20:15:41.102395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.102375 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5c46af/premium-w3-5c46af-kserve-5df64676bd-kss6f"] Nov 02 20:15:41.102484 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.102471 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.154505 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.154469 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz"] Nov 02 20:15:41.201999 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.201919 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" podStartSLOduration=2.201900689 podStartE2EDuration="2.201900689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:41.200688022 +0000 UTC m=+2847.149343872" watchObservedRunningTime="2026-11-02 20:15:41.201900689 +0000 UTC m=+2847.150556528" Nov 02 20:15:41.266936 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.266875 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.267173 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.267092 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94rw2\" (UniqueName: \"kubernetes.io/projected/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-kube-api-access-94rw2\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.267284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.267253 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.368688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.368601 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.368848 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.368719 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.368848 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.368753 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94rw2\" (UniqueName: \"kubernetes.io/projected/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-kube-api-access-94rw2\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.369448 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.369424 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.371062 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.371030 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.421320 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.421278 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94rw2\" (UniqueName: \"kubernetes.io/projected/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-kube-api-access-94rw2\") pod \"maas-api-revoke-keys-e2e-inf-b-19d134-wdffz\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.716404 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.716314 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:41.851809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.851772 4966 generic.go:356] "Generic (PLEG): container finished" podID="71b5ae16-6416-4ca5-8379-ef275bb98b4e" containerID="e7d010d367b39f1a0f4bc114fd947defe0bb1e18b184c1267ee7b271e5044a87" exitCode=0 Nov 02 20:15:41.852351 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:41.851925 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" event={"ID":"71b5ae16-6416-4ca5-8379-ef275bb98b4e","Type":"ContainerDied","Data":"e7d010d367b39f1a0f4bc114fd947defe0bb1e18b184c1267ee7b271e5044a87"} Nov 02 20:15:42.092011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:42.091972 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz"] Nov 02 20:15:42.092219 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:42.092190 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6659ca8e_f55f_4485_bb15_8ef83b44f6dc.slice/crio-f211701dbec8683d1f0b1a198892384b12213d232a556cf6ad00096424ded57b WatchSource:0}: Error finding container f211701dbec8683d1f0b1a198892384b12213d232a556cf6ad00096424ded57b: Status 404 returned error can't find the container with id f211701dbec8683d1f0b1a198892384b12213d232a556cf6ad00096424ded57b Nov 02 20:15:42.185173 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:42.185138 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fe8400-2d35-4658-a087-5a12feb990cc" path="/var/lib/kubelet/pods/04fe8400-2d35-4658-a087-5a12feb990cc/volumes" Nov 02 20:15:42.185659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:42.185642 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce95b19-3999-488e-9653-50055cf66632" path="/var/lib/kubelet/pods/dce95b19-3999-488e-9653-50055cf66632/volumes" Nov 02 20:15:42.863017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:42.862976 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" event={"ID":"6659ca8e-f55f-4485-bb15-8ef83b44f6dc","Type":"ContainerStarted","Data":"09db9f46674482a9d4ae616335ed87f17f9dec3207351eb3b4fc76793c5a8eec"} Nov 02 20:15:42.863575 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:42.863076 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" event={"ID":"6659ca8e-f55f-4485-bb15-8ef83b44f6dc","Type":"ContainerStarted","Data":"f211701dbec8683d1f0b1a198892384b12213d232a556cf6ad00096424ded57b"} Nov 02 20:15:42.981021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:42.980966 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" podStartSLOduration=1.98095081 podStartE2EDuration="1.98095081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:42.97798264 +0000 UTC m=+2848.926638480" watchObservedRunningTime="2026-11-02 20:15:42.98095081 +0000 UTC m=+2848.929606649" Nov 02 20:15:43.012775 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.012752 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:43.190197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.190116 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-xks-ca\") pod \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " Nov 02 20:15:43.190197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.190177 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-openshift-ca\") pod \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " Nov 02 20:15:43.190423 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.190372 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b5ae16-6416-4ca5-8379-ef275bb98b4e-kube-api-access-ldk9p\" (UniqueName: \"kubernetes.io/projected/71b5ae16-6416-4ca5-8379-ef275bb98b4e-kube-api-access-ldk9p\") pod \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\" (UID: \"71b5ae16-6416-4ca5-8379-ef275bb98b4e\") " Nov 02 20:15:43.190644 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.190615 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-openshift-ca" pod "71b5ae16-6416-4ca5-8379-ef275bb98b4e" (UID: "71b5ae16-6416-4ca5-8379-ef275bb98b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:43.190778 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.190763 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.192270 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.192253 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-xks-ca" pod "71b5ae16-6416-4ca5-8379-ef275bb98b4e" (UID: "71b5ae16-6416-4ca5-8379-ef275bb98b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:43.192421 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.192405 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b5ae16-6416-4ca5-8379-ef275bb98b4e-kube-api-access-ldk9p" pod "71b5ae16-6416-4ca5-8379-ef275bb98b4e" (UID: "71b5ae16-6416-4ca5-8379-ef275bb98b4e"). InnerVolumeSpecName "kube-api-access-ldk9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:43.291835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.291794 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldk9p\" (UniqueName: \"kubernetes.io/projected/71b5ae16-6416-4ca5-8379-ef275bb98b4e-kube-api-access-ldk9p\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.291835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.291835 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5ae16-6416-4ca5-8379-ef275bb98b4e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.871734 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.871695 4966 generic.go:356] "Generic (PLEG): container finished" podID="6659ca8e-f55f-4485-bb15-8ef83b44f6dc" containerID="09db9f46674482a9d4ae616335ed87f17f9dec3207351eb3b4fc76793c5a8eec" exitCode=0 Nov 02 20:15:43.872167 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.871776 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" event={"ID":"6659ca8e-f55f-4485-bb15-8ef83b44f6dc","Type":"ContainerDied","Data":"09db9f46674482a9d4ae616335ed87f17f9dec3207351eb3b4fc76793c5a8eec"} Nov 02 20:15:43.873275 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.873251 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" Nov 02 20:15:43.873557 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.873289 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq" event={"ID":"71b5ae16-6416-4ca5-8379-ef275bb98b4e","Type":"ContainerDied","Data":"5e5606935c221ce0e913534119d0ca55ade25ecb0d2d27e0b651df5b72478a22"} Nov 02 20:15:43.873557 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:43.873363 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e5606935c221ce0e913534119d0ca55ade25ecb0d2d27e0b651df5b72478a22" Nov 02 20:15:44.753359 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:15:44.753291 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:15:45.037676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.037650 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:45.213645 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.213611 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-xks-ca\") pod \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " Nov 02 20:15:45.213808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.213746 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-kube-api-access-94rw2\" (UniqueName: \"kubernetes.io/projected/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-kube-api-access-94rw2\") pod \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " Nov 02 20:15:45.213882 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.213867 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-openshift-ca\") pod \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\" (UID: \"6659ca8e-f55f-4485-bb15-8ef83b44f6dc\") " Nov 02 20:15:45.214248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.214219 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-openshift-ca" pod "6659ca8e-f55f-4485-bb15-8ef83b44f6dc" (UID: "6659ca8e-f55f-4485-bb15-8ef83b44f6dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:45.215865 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.215837 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-kube-api-access-94rw2" pod "6659ca8e-f55f-4485-bb15-8ef83b44f6dc" (UID: "6659ca8e-f55f-4485-bb15-8ef83b44f6dc"). InnerVolumeSpecName "kube-api-access-94rw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:45.215987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.215917 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-xks-ca" pod "6659ca8e-f55f-4485-bb15-8ef83b44f6dc" (UID: "6659ca8e-f55f-4485-bb15-8ef83b44f6dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:45.315046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.314951 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94rw2\" (UniqueName: \"kubernetes.io/projected/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-kube-api-access-94rw2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.315046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.314994 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.315046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.315009 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6659ca8e-f55f-4485-bb15-8ef83b44f6dc-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:45.889730 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.889703 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" Nov 02 20:15:45.889916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.889740 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz" event={"ID":"6659ca8e-f55f-4485-bb15-8ef83b44f6dc","Type":"ContainerDied","Data":"f211701dbec8683d1f0b1a198892384b12213d232a556cf6ad00096424ded57b"} Nov 02 20:15:45.889916 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:45.889796 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f211701dbec8683d1f0b1a198892384b12213d232a556cf6ad00096424ded57b" Nov 02 20:15:47.147568 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.147532 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r"] Nov 02 20:15:47.148336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.148316 4966 cpu_manager.go:414] "containerMap: removing container" podUID="71b5ae16-6416-4ca5-8379-ef275bb98b4e" containerName="revoke-keys" Nov 02 20:15:47.148414 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.148339 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b5ae16-6416-4ca5-8379-ef275bb98b4e" containerName="revoke-keys" Nov 02 20:15:47.148414 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.148398 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6659ca8e-f55f-4485-bb15-8ef83b44f6dc" containerName="revoke-keys" Nov 02 20:15:47.148414 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.148406 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6659ca8e-f55f-4485-bb15-8ef83b44f6dc" containerName="revoke-keys" Nov 02 20:15:47.148558 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.148546 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6659ca8e-f55f-4485-bb15-8ef83b44f6dc" containerName="revoke-keys" Nov 02 20:15:47.148596 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.148566 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b5ae16-6416-4ca5-8379-ef275bb98b4e" containerName="revoke-keys" Nov 02 20:15:47.152672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.152652 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.203278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.203245 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r"] Nov 02 20:15:47.245425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.245378 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:15:47.245606 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.245470 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:15:47.247271 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.247249 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:15:47.247426 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.247360 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:15:47.337103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.337065 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdpd\" (UniqueName: \"kubernetes.io/projected/92840f1b-d9f8-462c-af5b-2f9d26723e9d-kube-api-access-dvdpd\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.337705 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.337142 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.337705 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.337194 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.438357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.438223 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.438357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.438284 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.438616 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.438415 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdpd\" (UniqueName: \"kubernetes.io/projected/92840f1b-d9f8-462c-af5b-2f9d26723e9d-kube-api-access-dvdpd\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.439197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.439169 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.440791 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.440763 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.492090 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.492052 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdpd\" (UniqueName: \"kubernetes.io/projected/92840f1b-d9f8-462c-af5b-2f9d26723e9d-kube-api-access-dvdpd\") pod \"maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:47.768563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:47.768523 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:48.137473 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:48.137441 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r"] Nov 02 20:15:48.138951 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:15:48.138904 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92840f1b_d9f8_462c_af5b_2f9d26723e9d.slice/crio-99e1971aa8422eb3b1318ba90e73b38644b55c7d7fce2750e932b6fe4bad0afa WatchSource:0}: Error finding container 99e1971aa8422eb3b1318ba90e73b38644b55c7d7fce2750e932b6fe4bad0afa: Status 404 returned error can't find the container with id 99e1971aa8422eb3b1318ba90e73b38644b55c7d7fce2750e932b6fe4bad0afa Nov 02 20:15:48.911901 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:48.911861 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" event={"ID":"92840f1b-d9f8-462c-af5b-2f9d26723e9d","Type":"ContainerStarted","Data":"c14d8cd97997c20da5dc3baf442949b53047360feb2f148d5478917b9d58b6ed"} Nov 02 20:15:48.912322 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:48.911953 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" event={"ID":"92840f1b-d9f8-462c-af5b-2f9d26723e9d","Type":"ContainerStarted","Data":"99e1971aa8422eb3b1318ba90e73b38644b55c7d7fce2750e932b6fe4bad0afa"} Nov 02 20:15:49.019486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:49.019435 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" podStartSLOduration=2.019421288 podStartE2EDuration="2.019421288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:49.01720336 +0000 UTC m=+2854.965859248" watchObservedRunningTime="2026-11-02 20:15:49.019421288 +0000 UTC m=+2854.968077127" Nov 02 20:15:49.919693 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:49.919660 4966 generic.go:356] "Generic (PLEG): container finished" podID="92840f1b-d9f8-462c-af5b-2f9d26723e9d" containerID="c14d8cd97997c20da5dc3baf442949b53047360feb2f148d5478917b9d58b6ed" exitCode=0 Nov 02 20:15:49.920128 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:49.919744 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" event={"ID":"92840f1b-d9f8-462c-af5b-2f9d26723e9d","Type":"ContainerDied","Data":"c14d8cd97997c20da5dc3baf442949b53047360feb2f148d5478917b9d58b6ed"} Nov 02 20:15:51.090483 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.090456 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:51.181788 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.181690 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-openshift-ca\") pod \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " Nov 02 20:15:51.181974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.181804 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-xks-ca\") pod \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " Nov 02 20:15:51.181974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.181846 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92840f1b-d9f8-462c-af5b-2f9d26723e9d-kube-api-access-dvdpd\" (UniqueName: \"kubernetes.io/projected/92840f1b-d9f8-462c-af5b-2f9d26723e9d-kube-api-access-dvdpd\") pod \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\" (UID: \"92840f1b-d9f8-462c-af5b-2f9d26723e9d\") " Nov 02 20:15:51.182156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.182127 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-openshift-ca" pod "92840f1b-d9f8-462c-af5b-2f9d26723e9d" (UID: "92840f1b-d9f8-462c-af5b-2f9d26723e9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:51.182370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.182345 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:51.184203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.184176 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92840f1b-d9f8-462c-af5b-2f9d26723e9d-kube-api-access-dvdpd" pod "92840f1b-d9f8-462c-af5b-2f9d26723e9d" (UID: "92840f1b-d9f8-462c-af5b-2f9d26723e9d"). InnerVolumeSpecName "kube-api-access-dvdpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:51.184611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.184588 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-xks-ca" pod "92840f1b-d9f8-462c-af5b-2f9d26723e9d" (UID: "92840f1b-d9f8-462c-af5b-2f9d26723e9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:51.284016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.283968 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdpd\" (UniqueName: \"kubernetes.io/projected/92840f1b-d9f8-462c-af5b-2f9d26723e9d-kube-api-access-dvdpd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:51.284016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.284017 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92840f1b-d9f8-462c-af5b-2f9d26723e9d-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:51.935782 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.935754 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" Nov 02 20:15:51.935974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.935793 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r" event={"ID":"92840f1b-d9f8-462c-af5b-2f9d26723e9d","Type":"ContainerDied","Data":"99e1971aa8422eb3b1318ba90e73b38644b55c7d7fce2750e932b6fe4bad0afa"} Nov 02 20:15:51.935974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:51.935849 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e1971aa8422eb3b1318ba90e73b38644b55c7d7fce2750e932b6fe4bad0afa" Nov 02 20:15:58.953192 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:15:58.953131 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm"] Nov 02 20:16:00.330871 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.330833 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw"] Nov 02 20:16:00.331589 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.331566 4966 cpu_manager.go:414] "containerMap: removing container" podUID="92840f1b-d9f8-462c-af5b-2f9d26723e9d" containerName="revoke-keys" Nov 02 20:16:00.331702 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.331590 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92840f1b-d9f8-462c-af5b-2f9d26723e9d" containerName="revoke-keys" Nov 02 20:16:00.331762 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.331734 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="92840f1b-d9f8-462c-af5b-2f9d26723e9d" containerName="revoke-keys" Nov 02 20:16:00.335244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.335222 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.386015 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.385977 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.386335 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.385986 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg"] Nov 02 20:16:00.386335 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.386329 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.386541 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.386425 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4q2\" (UniqueName: \"kubernetes.io/projected/84274e96-6147-4b01-b9f6-4202e2c05641-kube-api-access-5v4q2\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.391201 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.391156 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.426877 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.426840 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:16:00.431421 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.431387 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw"] Nov 02 20:16:00.431685 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.431661 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:16:00.431767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.431686 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:16:00.431767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.431667 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:16:00.444955 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.444917 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg"] Nov 02 20:16:00.488088 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.488040 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.488284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.488185 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.488284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.488234 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmn7\" (UniqueName: \"kubernetes.io/projected/aa22eabd-1266-4a41-b3ab-e26b9c47722c-kube-api-access-plmn7\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.488428 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.488400 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.488512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.488497 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.488614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.488600 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4q2\" (UniqueName: \"kubernetes.io/projected/84274e96-6147-4b01-b9f6-4202e2c05641-kube-api-access-5v4q2\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.489449 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.489419 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.491033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.490997 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.541446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.541412 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4q2\" (UniqueName: \"kubernetes.io/projected/84274e96-6147-4b01-b9f6-4202e2c05641-kube-api-access-5v4q2\") pod \"maas-api-revoke-sub-037b901d-579cw\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.589809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.589612 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.589809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.589687 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.589809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.589716 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmn7\" (UniqueName: \"kubernetes.io/projected/aa22eabd-1266-4a41-b3ab-e26b9c47722c-kube-api-access-plmn7\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.591196 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.591166 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.592980 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.592955 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.643012 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.642977 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmn7\" (UniqueName: \"kubernetes.io/projected/aa22eabd-1266-4a41-b3ab-e26b9c47722c-kube-api-access-plmn7\") pod \"maas-api-revoke-sub-7bede2ba-d8xzg\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:00.648873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.648848 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:00.712935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:00.712887 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:01.027772 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:01.027738 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw"] Nov 02 20:16:01.028753 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:16:01.028722 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84274e96_6147_4b01_b9f6_4202e2c05641.slice/crio-cd2d9a0d2ccf7ca20f1abfa2163116f9dc6aa39ec96a1a16322178a5d25bb84f WatchSource:0}: Error finding container cd2d9a0d2ccf7ca20f1abfa2163116f9dc6aa39ec96a1a16322178a5d25bb84f: Status 404 returned error can't find the container with id cd2d9a0d2ccf7ca20f1abfa2163116f9dc6aa39ec96a1a16322178a5d25bb84f Nov 02 20:16:01.130403 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:01.130363 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg"] Nov 02 20:16:01.135981 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:16:01.135938 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa22eabd_1266_4a41_b3ab_e26b9c47722c.slice/crio-0603090505ebb1cd6be541d371d14d635483dfd208d8d55ae8d290de934f3bed WatchSource:0}: Error finding container 0603090505ebb1cd6be541d371d14d635483dfd208d8d55ae8d290de934f3bed: Status 404 returned error can't find the container with id 0603090505ebb1cd6be541d371d14d635483dfd208d8d55ae8d290de934f3bed Nov 02 20:16:02.030726 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:02.030690 4966 generic.go:356] "Generic (PLEG): container finished" podID="84274e96-6147-4b01-b9f6-4202e2c05641" containerID="ebadc432d5c4e551b3cabc7c9102a3c9fd094efdb89475444d323feb1d296e15" exitCode=0 Nov 02 20:16:02.031151 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:02.030911 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" event={"ID":"84274e96-6147-4b01-b9f6-4202e2c05641","Type":"ContainerDied","Data":"ebadc432d5c4e551b3cabc7c9102a3c9fd094efdb89475444d323feb1d296e15"} Nov 02 20:16:02.031151 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:02.030972 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" event={"ID":"84274e96-6147-4b01-b9f6-4202e2c05641","Type":"ContainerStarted","Data":"cd2d9a0d2ccf7ca20f1abfa2163116f9dc6aa39ec96a1a16322178a5d25bb84f"} Nov 02 20:16:02.033211 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:02.033184 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" event={"ID":"aa22eabd-1266-4a41-b3ab-e26b9c47722c","Type":"ContainerStarted","Data":"c598c9ea8b8dce92539fdd1f71b1b76384fdf5e978682278e5b7e7750b59f9ce"} Nov 02 20:16:02.033371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:02.033230 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" event={"ID":"aa22eabd-1266-4a41-b3ab-e26b9c47722c","Type":"ContainerStarted","Data":"0603090505ebb1cd6be541d371d14d635483dfd208d8d55ae8d290de934f3bed"} Nov 02 20:16:02.251717 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:02.251596 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" podStartSLOduration=2.251571428 podStartE2EDuration="2.251571428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:02.245996296 +0000 UTC m=+2868.194652136" watchObservedRunningTime="2026-11-02 20:16:02.251571428 +0000 UTC m=+2868.200227270" Nov 02 20:16:03.042920 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.042884 4966 generic.go:356] "Generic (PLEG): container finished" podID="aa22eabd-1266-4a41-b3ab-e26b9c47722c" containerID="c598c9ea8b8dce92539fdd1f71b1b76384fdf5e978682278e5b7e7750b59f9ce" exitCode=0 Nov 02 20:16:03.043432 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.042977 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" event={"ID":"aa22eabd-1266-4a41-b3ab-e26b9c47722c","Type":"ContainerDied","Data":"c598c9ea8b8dce92539fdd1f71b1b76384fdf5e978682278e5b7e7750b59f9ce"} Nov 02 20:16:03.213968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.213942 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:03.323536 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.323452 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-openshift-ca\") pod \"84274e96-6147-4b01-b9f6-4202e2c05641\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " Nov 02 20:16:03.323837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.323585 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-xks-ca\") pod \"84274e96-6147-4b01-b9f6-4202e2c05641\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " Nov 02 20:16:03.323837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.323615 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84274e96-6147-4b01-b9f6-4202e2c05641-kube-api-access-5v4q2\" (UniqueName: \"kubernetes.io/projected/84274e96-6147-4b01-b9f6-4202e2c05641-kube-api-access-5v4q2\") pod \"84274e96-6147-4b01-b9f6-4202e2c05641\" (UID: \"84274e96-6147-4b01-b9f6-4202e2c05641\") " Nov 02 20:16:03.323837 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.323823 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-openshift-ca" pod "84274e96-6147-4b01-b9f6-4202e2c05641" (UID: "84274e96-6147-4b01-b9f6-4202e2c05641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:03.324046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.324013 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:03.325843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.325814 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84274e96-6147-4b01-b9f6-4202e2c05641-kube-api-access-5v4q2" pod "84274e96-6147-4b01-b9f6-4202e2c05641" (UID: "84274e96-6147-4b01-b9f6-4202e2c05641"). InnerVolumeSpecName "kube-api-access-5v4q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:03.325987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.325927 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-xks-ca" pod "84274e96-6147-4b01-b9f6-4202e2c05641" (UID: "84274e96-6147-4b01-b9f6-4202e2c05641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:03.425133 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.425097 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v4q2\" (UniqueName: \"kubernetes.io/projected/84274e96-6147-4b01-b9f6-4202e2c05641-kube-api-access-5v4q2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:03.425133 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.425128 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84274e96-6147-4b01-b9f6-4202e2c05641-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:03.954224 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:03.954170 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" podUID="939030d5-3b39-4d34-9b74-f0d5a464415a" containerName="payload-pre-processing" containerID="cri-o://e762434b20a0df531a3974e3c4207932a3ab4d359745b4d1737974b43544b103" gracePeriod=25 Nov 02 20:16:04.054210 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.054177 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" Nov 02 20:16:04.054750 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.054204 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw" event={"ID":"84274e96-6147-4b01-b9f6-4202e2c05641","Type":"ContainerDied","Data":"cd2d9a0d2ccf7ca20f1abfa2163116f9dc6aa39ec96a1a16322178a5d25bb84f"} Nov 02 20:16:04.054750 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.054264 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2d9a0d2ccf7ca20f1abfa2163116f9dc6aa39ec96a1a16322178a5d25bb84f" Nov 02 20:16:04.199549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.199523 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:04.335077 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.335040 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-openshift-ca\") pod \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " Nov 02 20:16:04.335277 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.335128 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa22eabd-1266-4a41-b3ab-e26b9c47722c-kube-api-access-plmn7\" (UniqueName: \"kubernetes.io/projected/aa22eabd-1266-4a41-b3ab-e26b9c47722c-kube-api-access-plmn7\") pod \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " Nov 02 20:16:04.335277 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.335216 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-xks-ca\") pod \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\" (UID: \"aa22eabd-1266-4a41-b3ab-e26b9c47722c\") " Nov 02 20:16:04.335491 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.335468 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-openshift-ca" pod "aa22eabd-1266-4a41-b3ab-e26b9c47722c" (UID: "aa22eabd-1266-4a41-b3ab-e26b9c47722c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:04.335657 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.335636 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:04.337517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.337486 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-xks-ca" pod "aa22eabd-1266-4a41-b3ab-e26b9c47722c" (UID: "aa22eabd-1266-4a41-b3ab-e26b9c47722c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:04.337517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.337494 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa22eabd-1266-4a41-b3ab-e26b9c47722c-kube-api-access-plmn7" pod "aa22eabd-1266-4a41-b3ab-e26b9c47722c" (UID: "aa22eabd-1266-4a41-b3ab-e26b9c47722c"). InnerVolumeSpecName "kube-api-access-plmn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:04.436855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.436818 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa22eabd-1266-4a41-b3ab-e26b9c47722c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:04.436855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:04.436850 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmn7\" (UniqueName: \"kubernetes.io/projected/aa22eabd-1266-4a41-b3ab-e26b9c47722c-kube-api-access-plmn7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:05.063939 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:05.063904 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" Nov 02 20:16:05.063939 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:05.063929 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg" event={"ID":"aa22eabd-1266-4a41-b3ab-e26b9c47722c","Type":"ContainerDied","Data":"0603090505ebb1cd6be541d371d14d635483dfd208d8d55ae8d290de934f3bed"} Nov 02 20:16:05.064458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:05.063984 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0603090505ebb1cd6be541d371d14d635483dfd208d8d55ae8d290de934f3bed" Nov 02 20:16:12.615133 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:12.615089 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" podUID="939030d5-3b39-4d34-9b74-f0d5a464415a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:16:14.756137 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:16:14.756095 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:16:22.614913 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:22.614862 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" podUID="939030d5-3b39-4d34-9b74-f0d5a464415a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:16:24.219633 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.219605 4966 generic.go:356] "Generic (PLEG): container finished" podID="939030d5-3b39-4d34-9b74-f0d5a464415a" containerID="e762434b20a0df531a3974e3c4207932a3ab4d359745b4d1737974b43544b103" exitCode=0 Nov 02 20:16:24.220016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.219688 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" event={"ID":"939030d5-3b39-4d34-9b74-f0d5a464415a","Type":"ContainerDied","Data":"e762434b20a0df531a3974e3c4207932a3ab4d359745b4d1737974b43544b103"} Nov 02 20:16:24.244571 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.244542 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:16:24.280895 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.280839 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939030d5-3b39-4d34-9b74-f0d5a464415a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939030d5-3b39-4d34-9b74-f0d5a464415a-plugins-config-volume\") pod \"939030d5-3b39-4d34-9b74-f0d5a464415a\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " Nov 02 20:16:24.281081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.281033 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939030d5-3b39-4d34-9b74-f0d5a464415a-kube-api-access-mjq46\" (UniqueName: \"kubernetes.io/projected/939030d5-3b39-4d34-9b74-f0d5a464415a-kube-api-access-mjq46\") pod \"939030d5-3b39-4d34-9b74-f0d5a464415a\" (UID: \"939030d5-3b39-4d34-9b74-f0d5a464415a\") " Nov 02 20:16:24.281506 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.281472 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939030d5-3b39-4d34-9b74-f0d5a464415a-plugins-config-volume" pod "939030d5-3b39-4d34-9b74-f0d5a464415a" (UID: "939030d5-3b39-4d34-9b74-f0d5a464415a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:24.282055 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.282027 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939030d5-3b39-4d34-9b74-f0d5a464415a-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:24.284020 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.283991 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939030d5-3b39-4d34-9b74-f0d5a464415a-kube-api-access-mjq46" pod "939030d5-3b39-4d34-9b74-f0d5a464415a" (UID: "939030d5-3b39-4d34-9b74-f0d5a464415a"). InnerVolumeSpecName "kube-api-access-mjq46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:24.344477 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.344439 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r"] Nov 02 20:16:24.349357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.349329 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5b64d-c8p4r"] Nov 02 20:16:24.384444 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:24.384405 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjq46\" (UniqueName: \"kubernetes.io/projected/939030d5-3b39-4d34-9b74-f0d5a464415a-kube-api-access-mjq46\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:25.228370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:25.228334 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" event={"ID":"939030d5-3b39-4d34-9b74-f0d5a464415a","Type":"ContainerDied","Data":"e2244e003b55b3964a48545b258ed82b693326b94dbbde3ffe59e110afbf1abc"} Nov 02 20:16:25.228370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:25.228359 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm" Nov 02 20:16:25.228869 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:25.228406 4966 scope.go:122] "RemoveContainer" containerID="e762434b20a0df531a3974e3c4207932a3ab4d359745b4d1737974b43544b103" Nov 02 20:16:25.382804 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:25.382766 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm"] Nov 02 20:16:25.386596 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:25.386565 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5b64d-7958f57699-w8dsm"] Nov 02 20:16:25.560370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:25.560248 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq"] Nov 02 20:16:25.563362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:25.563330 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28fa24-q2rjq"] Nov 02 20:16:26.183081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:26.183045 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b5ae16-6416-4ca5-8379-ef275bb98b4e" path="/var/lib/kubelet/pods/71b5ae16-6416-4ca5-8379-ef275bb98b4e/volumes" Nov 02 20:16:26.183580 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:26.183558 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92840f1b-d9f8-462c-af5b-2f9d26723e9d" path="/var/lib/kubelet/pods/92840f1b-d9f8-462c-af5b-2f9d26723e9d/volumes" Nov 02 20:16:26.183980 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:26.183959 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939030d5-3b39-4d34-9b74-f0d5a464415a" path="/var/lib/kubelet/pods/939030d5-3b39-4d34-9b74-f0d5a464415a/volumes" Nov 02 20:16:31.297701 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.297662 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz"] Nov 02 20:16:31.343695 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.343658 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19d134-wdffz"] Nov 02 20:16:31.707133 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707046 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt"] Nov 02 20:16:31.707619 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707603 4966 cpu_manager.go:414] "containerMap: removing container" podUID="aa22eabd-1266-4a41-b3ab-e26b9c47722c" containerName="revoke-keys" Nov 02 20:16:31.707672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707620 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa22eabd-1266-4a41-b3ab-e26b9c47722c" containerName="revoke-keys" Nov 02 20:16:31.707672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707643 4966 cpu_manager.go:414] "containerMap: removing container" podUID="939030d5-3b39-4d34-9b74-f0d5a464415a" containerName="payload-pre-processing" Nov 02 20:16:31.707672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707649 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939030d5-3b39-4d34-9b74-f0d5a464415a" containerName="payload-pre-processing" Nov 02 20:16:31.707672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707658 4966 cpu_manager.go:414] "containerMap: removing container" podUID="84274e96-6147-4b01-b9f6-4202e2c05641" containerName="revoke-keys" Nov 02 20:16:31.707672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707664 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84274e96-6147-4b01-b9f6-4202e2c05641" containerName="revoke-keys" Nov 02 20:16:31.707846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707741 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa22eabd-1266-4a41-b3ab-e26b9c47722c" containerName="revoke-keys" Nov 02 20:16:31.707846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707751 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="939030d5-3b39-4d34-9b74-f0d5a464415a" containerName="payload-pre-processing" Nov 02 20:16:31.707846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.707757 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="84274e96-6147-4b01-b9f6-4202e2c05641" containerName="revoke-keys" Nov 02 20:16:31.712563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.712537 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.761757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.761721 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:16:31.762009 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.761836 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:16:31.762009 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.761865 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:16:31.762009 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.761923 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:16:31.765672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.765640 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.765840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.765719 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt"] Nov 02 20:16:31.765840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.765731 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4qx\" (UniqueName: \"kubernetes.io/projected/c85da80c-7079-438e-9c07-520561b8b6c3-kube-api-access-kk4qx\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.765840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.765794 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.867142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.867099 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.867390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.867211 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4qx\" (UniqueName: \"kubernetes.io/projected/c85da80c-7079-438e-9c07-520561b8b6c3-kube-api-access-kk4qx\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.867390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.867263 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.868144 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.868117 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.869758 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.869736 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:31.919967 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:31.919930 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4qx\" (UniqueName: \"kubernetes.io/projected/c85da80c-7079-438e-9c07-520561b8b6c3-kube-api-access-kk4qx\") pod \"maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:32.026598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:32.026560 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:32.182789 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:32.182748 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6659ca8e-f55f-4485-bb15-8ef83b44f6dc" path="/var/lib/kubelet/pods/6659ca8e-f55f-4485-bb15-8ef83b44f6dc/volumes" Nov 02 20:16:32.407014 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:32.406975 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt"] Nov 02 20:16:32.407796 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:16:32.407769 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85da80c_7079_438e_9c07_520561b8b6c3.slice/crio-88afd1a0389609ba5ffd15eeb863ddd9fddc8d9326a5576b62c7f0c1abe5ac04 WatchSource:0}: Error finding container 88afd1a0389609ba5ffd15eeb863ddd9fddc8d9326a5576b62c7f0c1abe5ac04: Status 404 returned error can't find the container with id 88afd1a0389609ba5ffd15eeb863ddd9fddc8d9326a5576b62c7f0c1abe5ac04 Nov 02 20:16:33.295679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.295642 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" event={"ID":"c85da80c-7079-438e-9c07-520561b8b6c3","Type":"ContainerStarted","Data":"9480149b27128365157159b34cc53864d66bac7bdd791e1cb79a5024636d6a4b"} Nov 02 20:16:33.295873 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.295698 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" event={"ID":"c85da80c-7079-438e-9c07-520561b8b6c3","Type":"ContainerStarted","Data":"88afd1a0389609ba5ffd15eeb863ddd9fddc8d9326a5576b62c7f0c1abe5ac04"} Nov 02 20:16:33.401660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.401609 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" podStartSLOduration=2.401593173 podStartE2EDuration="2.401593173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:33.399960496 +0000 UTC m=+2899.348616335" watchObservedRunningTime="2026-11-02 20:16:33.401593173 +0000 UTC m=+2899.350249012" Nov 02 20:16:33.901349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.901314 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz"] Nov 02 20:16:33.905282 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.905265 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:33.958942 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.958908 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz"] Nov 02 20:16:33.995275 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.995244 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:33.995498 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.995307 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:33.995498 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:33.995462 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcvs\" (UniqueName: \"kubernetes.io/projected/c64ef426-a7e2-4a6c-9ef1-655c960ded11-kube-api-access-hfcvs\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.096597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.096560 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcvs\" (UniqueName: \"kubernetes.io/projected/c64ef426-a7e2-4a6c-9ef1-655c960ded11-kube-api-access-hfcvs\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.096791 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.096680 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.096791 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.096705 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.097391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.097360 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.099171 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.099147 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.149359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.149290 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcvs\" (UniqueName: \"kubernetes.io/projected/c64ef426-a7e2-4a6c-9ef1-655c960ded11-kube-api-access-hfcvs\") pod \"maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.218592 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.218568 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:34.305497 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.305457 4966 generic.go:356] "Generic (PLEG): container finished" podID="c85da80c-7079-438e-9c07-520561b8b6c3" containerID="9480149b27128365157159b34cc53864d66bac7bdd791e1cb79a5024636d6a4b" exitCode=0 Nov 02 20:16:34.305721 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.305700 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" event={"ID":"c85da80c-7079-438e-9c07-520561b8b6c3","Type":"ContainerDied","Data":"9480149b27128365157159b34cc53864d66bac7bdd791e1cb79a5024636d6a4b"} Nov 02 20:16:34.589493 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:34.589462 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz"] Nov 02 20:16:34.590400 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:16:34.590376 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64ef426_a7e2_4a6c_9ef1_655c960ded11.slice/crio-4f9b93567310d82b6538e89882104bcfc712f971cb72891168b42fd34f764d92 WatchSource:0}: Error finding container 4f9b93567310d82b6538e89882104bcfc712f971cb72891168b42fd34f764d92: Status 404 returned error can't find the container with id 4f9b93567310d82b6538e89882104bcfc712f971cb72891168b42fd34f764d92 Nov 02 20:16:35.316137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.316094 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" event={"ID":"c64ef426-a7e2-4a6c-9ef1-655c960ded11","Type":"ContainerStarted","Data":"cceb8a99af76b6c59b51d90f2f3e7706f1116bcb4ea44cb0752373b3d416f1ad"} Nov 02 20:16:35.316741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.316159 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" event={"ID":"c64ef426-a7e2-4a6c-9ef1-655c960ded11","Type":"ContainerStarted","Data":"4f9b93567310d82b6538e89882104bcfc712f971cb72891168b42fd34f764d92"} Nov 02 20:16:35.425492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.425432 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" podStartSLOduration=2.425420129 podStartE2EDuration="2.425420129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:35.423569978 +0000 UTC m=+2901.372225843" watchObservedRunningTime="2026-11-02 20:16:35.425420129 +0000 UTC m=+2901.374075968" Nov 02 20:16:35.468036 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.468006 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:35.614460 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.614365 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85da80c-7079-438e-9c07-520561b8b6c3-kube-api-access-kk4qx\" (UniqueName: \"kubernetes.io/projected/c85da80c-7079-438e-9c07-520561b8b6c3-kube-api-access-kk4qx\") pod \"c85da80c-7079-438e-9c07-520561b8b6c3\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " Nov 02 20:16:35.614460 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.614435 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-openshift-ca\") pod \"c85da80c-7079-438e-9c07-520561b8b6c3\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " Nov 02 20:16:35.614718 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.614500 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-xks-ca\") pod \"c85da80c-7079-438e-9c07-520561b8b6c3\" (UID: \"c85da80c-7079-438e-9c07-520561b8b6c3\") " Nov 02 20:16:35.615354 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.615287 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-openshift-ca" pod "c85da80c-7079-438e-9c07-520561b8b6c3" (UID: "c85da80c-7079-438e-9c07-520561b8b6c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:35.617185 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.617151 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-xks-ca" pod "c85da80c-7079-438e-9c07-520561b8b6c3" (UID: "c85da80c-7079-438e-9c07-520561b8b6c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:35.618127 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.618087 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85da80c-7079-438e-9c07-520561b8b6c3-kube-api-access-kk4qx" pod "c85da80c-7079-438e-9c07-520561b8b6c3" (UID: "c85da80c-7079-438e-9c07-520561b8b6c3"). InnerVolumeSpecName "kube-api-access-kk4qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:35.716495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.716447 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.716495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.716491 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4qx\" (UniqueName: \"kubernetes.io/projected/c85da80c-7079-438e-9c07-520561b8b6c3-kube-api-access-kk4qx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:35.716495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:35.716502 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85da80c-7079-438e-9c07-520561b8b6c3-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:36.328582 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:36.328546 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" Nov 02 20:16:36.328582 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:36.328568 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt" event={"ID":"c85da80c-7079-438e-9c07-520561b8b6c3","Type":"ContainerDied","Data":"88afd1a0389609ba5ffd15eeb863ddd9fddc8d9326a5576b62c7f0c1abe5ac04"} Nov 02 20:16:36.329099 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:36.328624 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88afd1a0389609ba5ffd15eeb863ddd9fddc8d9326a5576b62c7f0c1abe5ac04" Nov 02 20:16:36.330683 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:36.330660 4966 generic.go:356] "Generic (PLEG): container finished" podID="c64ef426-a7e2-4a6c-9ef1-655c960ded11" containerID="cceb8a99af76b6c59b51d90f2f3e7706f1116bcb4ea44cb0752373b3d416f1ad" exitCode=0 Nov 02 20:16:36.330850 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:36.330751 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" event={"ID":"c64ef426-a7e2-4a6c-9ef1-655c960ded11","Type":"ContainerDied","Data":"cceb8a99af76b6c59b51d90f2f3e7706f1116bcb4ea44cb0752373b3d416f1ad"} Nov 02 20:16:37.492386 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.492360 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:37.635693 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.635613 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-openshift-ca\") pod \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " Nov 02 20:16:37.635864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.635696 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-xks-ca\") pod \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " Nov 02 20:16:37.635864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.635824 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64ef426-a7e2-4a6c-9ef1-655c960ded11-kube-api-access-hfcvs\" (UniqueName: \"kubernetes.io/projected/c64ef426-a7e2-4a6c-9ef1-655c960ded11-kube-api-access-hfcvs\") pod \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\" (UID: \"c64ef426-a7e2-4a6c-9ef1-655c960ded11\") " Nov 02 20:16:37.636062 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.636037 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-openshift-ca" pod "c64ef426-a7e2-4a6c-9ef1-655c960ded11" (UID: "c64ef426-a7e2-4a6c-9ef1-655c960ded11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:37.636357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.636336 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:37.637840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.637814 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-xks-ca" pod "c64ef426-a7e2-4a6c-9ef1-655c960ded11" (UID: "c64ef426-a7e2-4a6c-9ef1-655c960ded11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:37.638030 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.638015 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ef426-a7e2-4a6c-9ef1-655c960ded11-kube-api-access-hfcvs" pod "c64ef426-a7e2-4a6c-9ef1-655c960ded11" (UID: "c64ef426-a7e2-4a6c-9ef1-655c960ded11"). InnerVolumeSpecName "kube-api-access-hfcvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:37.737143 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.737101 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfcvs\" (UniqueName: \"kubernetes.io/projected/c64ef426-a7e2-4a6c-9ef1-655c960ded11-kube-api-access-hfcvs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:37.737143 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:37.737138 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64ef426-a7e2-4a6c-9ef1-655c960ded11-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:38.348519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.348485 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" Nov 02 20:16:38.348519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.348501 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz" event={"ID":"c64ef426-a7e2-4a6c-9ef1-655c960ded11","Type":"ContainerDied","Data":"4f9b93567310d82b6538e89882104bcfc712f971cb72891168b42fd34f764d92"} Nov 02 20:16:38.348739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.348559 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f9b93567310d82b6538e89882104bcfc712f971cb72891168b42fd34f764d92" Nov 02 20:16:38.449244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.449141 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd"] Nov 02 20:16:38.452651 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.452621 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw"] Nov 02 20:16:38.817836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.817795 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8"] Nov 02 20:16:38.818489 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.818469 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c85da80c-7079-438e-9c07-520561b8b6c3" containerName="revoke-keys" Nov 02 20:16:38.818552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.818490 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85da80c-7079-438e-9c07-520561b8b6c3" containerName="revoke-keys" Nov 02 20:16:38.818552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.818535 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c64ef426-a7e2-4a6c-9ef1-655c960ded11" containerName="revoke-keys" Nov 02 20:16:38.818552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.818541 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ef426-a7e2-4a6c-9ef1-655c960ded11" containerName="revoke-keys" Nov 02 20:16:38.818664 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.818618 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ef426-a7e2-4a6c-9ef1-655c960ded11" containerName="revoke-keys" Nov 02 20:16:38.818664 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.818628 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85da80c-7079-438e-9c07-520561b8b6c3" containerName="revoke-keys" Nov 02 20:16:38.822023 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.822004 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:38.870941 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.870901 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:16:38.871505 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.871474 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:16:38.871672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.871549 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:16:38.871842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.871824 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:16:38.875709 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.875223 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8"] Nov 02 20:16:38.951096 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.951043 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjwwc\" (UniqueName: \"kubernetes.io/projected/e530f339-18f4-44d0-85f2-33f984e993d3-kube-api-access-rjwwc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:38.951328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.951168 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:38.951328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:38.951269 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.052257 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.052218 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.052483 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.052373 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjwwc\" (UniqueName: \"kubernetes.io/projected/e530f339-18f4-44d0-85f2-33f984e993d3-kube-api-access-rjwwc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.052483 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.052437 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.053158 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.053120 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.055458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.055433 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.106137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.106052 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjwwc\" (UniqueName: \"kubernetes.io/projected/e530f339-18f4-44d0-85f2-33f984e993d3-kube-api-access-rjwwc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.136025 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.135984 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:39.512584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:39.512555 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8"] Nov 02 20:16:39.513423 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:16:39.513397 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode530f339_18f4_44d0_85f2_33f984e993d3.slice/crio-b214309fabadd6814ff225625fba827a0345506a66acb5e50e23b14ae7a6376f WatchSource:0}: Error finding container b214309fabadd6814ff225625fba827a0345506a66acb5e50e23b14ae7a6376f: Status 404 returned error can't find the container with id b214309fabadd6814ff225625fba827a0345506a66acb5e50e23b14ae7a6376f Nov 02 20:16:40.363640 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:40.363601 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" event={"ID":"e530f339-18f4-44d0-85f2-33f984e993d3","Type":"ContainerStarted","Data":"6db63625c1d5bde8d581f197c98200a8cf467149d9bb5c06e37ef43ae13b5970"} Nov 02 20:16:40.364046 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:40.363696 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" event={"ID":"e530f339-18f4-44d0-85f2-33f984e993d3","Type":"ContainerStarted","Data":"b214309fabadd6814ff225625fba827a0345506a66acb5e50e23b14ae7a6376f"} Nov 02 20:16:40.470864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:40.470811 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" podStartSLOduration=2.4707969260000002 podStartE2EDuration="2.470796926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:40.470011843 +0000 UTC m=+2906.418667684" watchObservedRunningTime="2026-11-02 20:16:40.470796926 +0000 UTC m=+2906.419452765" Nov 02 20:16:41.372640 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:41.372603 4966 generic.go:356] "Generic (PLEG): container finished" podID="e530f339-18f4-44d0-85f2-33f984e993d3" containerID="6db63625c1d5bde8d581f197c98200a8cf467149d9bb5c06e37ef43ae13b5970" exitCode=0 Nov 02 20:16:41.373107 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:41.372688 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" event={"ID":"e530f339-18f4-44d0-85f2-33f984e993d3","Type":"ContainerDied","Data":"6db63625c1d5bde8d581f197c98200a8cf467149d9bb5c06e37ef43ae13b5970"} Nov 02 20:16:42.527830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.527799 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:42.692635 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.692542 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-openshift-ca\") pod \"e530f339-18f4-44d0-85f2-33f984e993d3\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " Nov 02 20:16:42.692824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.692650 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-xks-ca\") pod \"e530f339-18f4-44d0-85f2-33f984e993d3\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " Nov 02 20:16:42.692824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.692739 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e530f339-18f4-44d0-85f2-33f984e993d3-kube-api-access-rjwwc\" (UniqueName: \"kubernetes.io/projected/e530f339-18f4-44d0-85f2-33f984e993d3-kube-api-access-rjwwc\") pod \"e530f339-18f4-44d0-85f2-33f984e993d3\" (UID: \"e530f339-18f4-44d0-85f2-33f984e993d3\") " Nov 02 20:16:42.692966 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.692937 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-openshift-ca" pod "e530f339-18f4-44d0-85f2-33f984e993d3" (UID: "e530f339-18f4-44d0-85f2-33f984e993d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:42.693155 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.693133 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:42.694762 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.694732 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-xks-ca" pod "e530f339-18f4-44d0-85f2-33f984e993d3" (UID: "e530f339-18f4-44d0-85f2-33f984e993d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:42.694881 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.694863 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e530f339-18f4-44d0-85f2-33f984e993d3-kube-api-access-rjwwc" pod "e530f339-18f4-44d0-85f2-33f984e993d3" (UID: "e530f339-18f4-44d0-85f2-33f984e993d3"). InnerVolumeSpecName "kube-api-access-rjwwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:42.793987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.793949 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e530f339-18f4-44d0-85f2-33f984e993d3-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:42.793987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:42.793980 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjwwc\" (UniqueName: \"kubernetes.io/projected/e530f339-18f4-44d0-85f2-33f984e993d3-kube-api-access-rjwwc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:43.375699 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:43.375662 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs"] Nov 02 20:16:43.387768 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:43.387735 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" event={"ID":"e530f339-18f4-44d0-85f2-33f984e993d3","Type":"ContainerDied","Data":"b214309fabadd6814ff225625fba827a0345506a66acb5e50e23b14ae7a6376f"} Nov 02 20:16:43.387958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:43.387794 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b214309fabadd6814ff225625fba827a0345506a66acb5e50e23b14ae7a6376f" Nov 02 20:16:43.387958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:43.387744 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8" Nov 02 20:16:44.753520 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:16:44.753474 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:16:45.864789 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:45.864741 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc"] Nov 02 20:16:48.376069 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:48.375996 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" podUID="8754ec15-5b0b-4188-9f35-4d00a8b24043" containerName="payload-pre-processing" containerID="cri-o://57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc" gracePeriod=25 Nov 02 20:16:50.866284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:50.866199 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" podUID="d7499b11-1af4-4ca5-acef-68a5d6dcde9b" containerName="payload-processing" containerID="cri-o://1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4" gracePeriod=25 Nov 02 20:16:51.137437 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.137408 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 20:16:51.186777 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.186745 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-kube-api-access-grffb\" (UniqueName: \"kubernetes.io/projected/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-kube-api-access-grffb\") pod \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " Nov 02 20:16:51.186970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.186827 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-plugins-config-volume\") pod \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\" (UID: \"d7499b11-1af4-4ca5-acef-68a5d6dcde9b\") " Nov 02 20:16:51.187176 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.187152 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-plugins-config-volume" pod "d7499b11-1af4-4ca5-acef-68a5d6dcde9b" (UID: "d7499b11-1af4-4ca5-acef-68a5d6dcde9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:51.189144 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.189121 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-kube-api-access-grffb" pod "d7499b11-1af4-4ca5-acef-68a5d6dcde9b" (UID: "d7499b11-1af4-4ca5-acef-68a5d6dcde9b"). InnerVolumeSpecName "kube-api-access-grffb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:51.288159 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.288105 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grffb\" (UniqueName: \"kubernetes.io/projected/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-kube-api-access-grffb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:51.288159 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.288149 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7499b11-1af4-4ca5-acef-68a5d6dcde9b-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:51.404635 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.404549 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr"] Nov 02 20:16:51.407637 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.407608 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5c46af-nl5dr"] Nov 02 20:16:51.457445 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.457411 4966 generic.go:356] "Generic (PLEG): container finished" podID="d7499b11-1af4-4ca5-acef-68a5d6dcde9b" containerID="1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4" exitCode=0 Nov 02 20:16:51.457642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.457467 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" Nov 02 20:16:51.457642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.457496 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" event={"ID":"d7499b11-1af4-4ca5-acef-68a5d6dcde9b","Type":"ContainerDied","Data":"1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4"} Nov 02 20:16:51.457642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.457558 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc" event={"ID":"d7499b11-1af4-4ca5-acef-68a5d6dcde9b","Type":"ContainerDied","Data":"cdcb401c3d16fd35eeba0807702150c6e2ee514ba187ed58118bea5cbbd00a83"} Nov 02 20:16:51.457642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.457588 4966 scope.go:122] "RemoveContainer" containerID="1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4" Nov 02 20:16:51.470217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.470194 4966 scope.go:122] "RemoveContainer" containerID="1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4" Nov 02 20:16:51.470534 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:16:51.470512 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4\": container with ID starting with 1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4 not found: ID does not exist" containerID="1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4" Nov 02 20:16:51.470643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.470540 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4"} err="failed to get container status \"1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4\": rpc error: code = NotFound desc = could not find container \"1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4\": container with ID starting with 1e963c3336d59126ed2c5d47be8aca3a9cefbdd625932903888737ae718013f4 not found: ID does not exist" Nov 02 20:16:51.610308 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.610263 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc"] Nov 02 20:16:51.615042 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:51.615013 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-43ceaa-5cbc8b8cfc-nrxgc"] Nov 02 20:16:52.183868 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:52.183836 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a796d742-fd83-43aa-9cae-a34052c4d20c" path="/var/lib/kubelet/pods/a796d742-fd83-43aa-9cae-a34052c4d20c/volumes" Nov 02 20:16:52.184272 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:52.184257 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7499b11-1af4-4ca5-acef-68a5d6dcde9b" path="/var/lib/kubelet/pods/d7499b11-1af4-4ca5-acef-68a5d6dcde9b/volumes" Nov 02 20:16:53.472432 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.472390 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" containerID="cri-o://499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4" gracePeriod=45 Nov 02 20:16:53.475691 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.475648 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" containerID="cri-o://251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a" gracePeriod=45 Nov 02 20:16:53.748508 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.748481 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 20:16:53.775990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.775966 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 20:16:53.818009 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.817974 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-model-cache\") pod \"378e604c-6be5-487a-a55a-669b6c22f9aa\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " Nov 02 20:16:53.818203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818057 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91074226-875a-4294-80f9-d761498e387a-tls-certs\" (UniqueName: \"kubernetes.io/secret/91074226-875a-4294-80f9-d761498e387a-tls-certs\") pod \"91074226-875a-4294-80f9-d761498e387a\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " Nov 02 20:16:53.818203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818089 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-home\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-home\") pod \"378e604c-6be5-487a-a55a-669b6c22f9aa\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " Nov 02 20:16:53.818203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818119 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378e604c-6be5-487a-a55a-669b6c22f9aa-kube-api-access-tsz6q\" (UniqueName: \"kubernetes.io/projected/378e604c-6be5-487a-a55a-669b6c22f9aa-kube-api-access-tsz6q\") pod \"378e604c-6be5-487a-a55a-669b6c22f9aa\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " Nov 02 20:16:53.818203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818157 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-model-cache\") pod \"91074226-875a-4294-80f9-d761498e387a\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " Nov 02 20:16:53.818203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818189 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-tmp-dir\") pod \"378e604c-6be5-487a-a55a-669b6c22f9aa\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818218 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-dshm\") pod \"91074226-875a-4294-80f9-d761498e387a\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818294 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-model-cache" pod "378e604c-6be5-487a-a55a-669b6c22f9aa" (UID: "378e604c-6be5-487a-a55a-669b6c22f9aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818331 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91074226-875a-4294-80f9-d761498e387a-kube-api-access-76mpd\" (UniqueName: \"kubernetes.io/projected/91074226-875a-4294-80f9-d761498e387a-kube-api-access-76mpd\") pod \"91074226-875a-4294-80f9-d761498e387a\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818371 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378e604c-6be5-487a-a55a-669b6c22f9aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/378e604c-6be5-487a-a55a-669b6c22f9aa-tls-certs\") pod \"378e604c-6be5-487a-a55a-669b6c22f9aa\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818401 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-home" pod "378e604c-6be5-487a-a55a-669b6c22f9aa" (UID: "378e604c-6be5-487a-a55a-669b6c22f9aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818434 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-dshm\") pod \"378e604c-6be5-487a-a55a-669b6c22f9aa\" (UID: \"378e604c-6be5-487a-a55a-669b6c22f9aa\") " Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818499 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-home\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-home\") pod \"91074226-875a-4294-80f9-d761498e387a\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " Nov 02 20:16:53.818542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818542 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-tmp-dir\") pod \"91074226-875a-4294-80f9-d761498e387a\" (UID: \"91074226-875a-4294-80f9-d761498e387a\") " Nov 02 20:16:53.819050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818953 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.819050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.818975 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.819246 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.819223 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-tmp-dir" pod "91074226-875a-4294-80f9-d761498e387a" (UID: "91074226-875a-4294-80f9-d761498e387a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.819511 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.819396 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-model-cache" pod "91074226-875a-4294-80f9-d761498e387a" (UID: "91074226-875a-4294-80f9-d761498e387a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.819511 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.819495 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-home" pod "91074226-875a-4294-80f9-d761498e387a" (UID: "91074226-875a-4294-80f9-d761498e387a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.819511 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.819619 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-tmp-dir" pod "378e604c-6be5-487a-a55a-669b6c22f9aa" (UID: "378e604c-6be5-487a-a55a-669b6c22f9aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.821049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.821027 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378e604c-6be5-487a-a55a-669b6c22f9aa-kube-api-access-tsz6q" pod "378e604c-6be5-487a-a55a-669b6c22f9aa" (UID: "378e604c-6be5-487a-a55a-669b6c22f9aa"). InnerVolumeSpecName "kube-api-access-tsz6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:53.821601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.821571 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-dshm" pod "91074226-875a-4294-80f9-d761498e387a" (UID: "91074226-875a-4294-80f9-d761498e387a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.821706 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.821641 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91074226-875a-4294-80f9-d761498e387a-tls-certs" pod "91074226-875a-4294-80f9-d761498e387a" (UID: "91074226-875a-4294-80f9-d761498e387a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:53.821998 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.821965 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-dshm" pod "378e604c-6be5-487a-a55a-669b6c22f9aa" (UID: "378e604c-6be5-487a-a55a-669b6c22f9aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:53.822243 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.822216 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91074226-875a-4294-80f9-d761498e387a-kube-api-access-76mpd" pod "91074226-875a-4294-80f9-d761498e387a" (UID: "91074226-875a-4294-80f9-d761498e387a"). InnerVolumeSpecName "kube-api-access-76mpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:53.822330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.822256 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378e604c-6be5-487a-a55a-669b6c22f9aa-tls-certs" pod "378e604c-6be5-487a-a55a-669b6c22f9aa" (UID: "378e604c-6be5-487a-a55a-669b6c22f9aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:53.920512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920482 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91074226-875a-4294-80f9-d761498e387a-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920511 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsz6q\" (UniqueName: \"kubernetes.io/projected/378e604c-6be5-487a-a55a-669b6c22f9aa-kube-api-access-tsz6q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920512 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920522 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920531 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920540 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920549 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76mpd\" (UniqueName: \"kubernetes.io/projected/91074226-875a-4294-80f9-d761498e387a-kube-api-access-76mpd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920557 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/378e604c-6be5-487a-a55a-669b6c22f9aa-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920567 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/378e604c-6be5-487a-a55a-669b6c22f9aa-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920574 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.920757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:53.920582 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91074226-875a-4294-80f9-d761498e387a-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:54.481570 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.481539 4966 generic.go:356] "Generic (PLEG): container finished" podID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerID="499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4" exitCode=0 Nov 02 20:16:54.482067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.481592 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" Nov 02 20:16:54.482067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.481625 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" event={"ID":"378e604c-6be5-487a-a55a-669b6c22f9aa","Type":"ContainerDied","Data":"499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4"} Nov 02 20:16:54.482067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.481671 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd" event={"ID":"378e604c-6be5-487a-a55a-669b6c22f9aa","Type":"ContainerDied","Data":"71b8ee9de05543cba8a7538ee87225d7ee38acccf78ce5d6477e97f18bf29669"} Nov 02 20:16:54.482067 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.481690 4966 scope.go:122] "RemoveContainer" containerID="499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4" Nov 02 20:16:54.483435 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.483411 4966 generic.go:356] "Generic (PLEG): container finished" podID="91074226-875a-4294-80f9-d761498e387a" containerID="251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a" exitCode=0 Nov 02 20:16:54.483540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.483452 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" event={"ID":"91074226-875a-4294-80f9-d761498e387a","Type":"ContainerDied","Data":"251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a"} Nov 02 20:16:54.483540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.483486 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" event={"ID":"91074226-875a-4294-80f9-d761498e387a","Type":"ContainerDied","Data":"b7140f2510ad20abeb7741d31991c3cb96aad74da4d9b836a2843c9fa7e4b97c"} Nov 02 20:16:54.483540 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.483457 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw" Nov 02 20:16:54.505966 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.504469 4966 scope.go:122] "RemoveContainer" containerID="499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4" Nov 02 20:16:54.505966 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:16:54.504959 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4\": container with ID starting with 499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4 not found: ID does not exist" containerID="499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4" Nov 02 20:16:54.505966 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.505001 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4"} err="failed to get container status \"499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4\": rpc error: code = NotFound desc = could not find container \"499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4\": container with ID starting with 499db469c1da8e79a94c67245bdf33ebaf95fcbe13c3bad09971ab193bb896c4 not found: ID does not exist" Nov 02 20:16:54.505966 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.505029 4966 scope.go:122] "RemoveContainer" containerID="251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a" Nov 02 20:16:54.514988 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.514967 4966 scope.go:122] "RemoveContainer" containerID="251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a" Nov 02 20:16:54.515321 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:16:54.515279 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a\": container with ID starting with 251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a not found: ID does not exist" containerID="251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a" Nov 02 20:16:54.515386 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.515334 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a"} err="failed to get container status \"251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a\": rpc error: code = NotFound desc = could not find container \"251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a\": container with ID starting with 251f88ec4aa2374cee5a7c4d1e0ee83b1bed2547078f9bb5124cacfd2b7de21a not found: ID does not exist" Nov 02 20:16:54.634812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.634771 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd"] Nov 02 20:16:54.637826 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.637797 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/distinct-2-w0-43ceaa-kserve-85bbbb7489-dhlkd"] Nov 02 20:16:54.737823 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.737732 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw"] Nov 02 20:16:54.743561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:54.743531 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-43ceaa/simulator-w0-43ceaa-kserve-8f9db4b7d-ht7mw"] Nov 02 20:16:55.917912 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:55.917875 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" podUID="8754ec15-5b0b-4188-9f35-4d00a8b24043" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:16:56.182621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:56.182540 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" path="/var/lib/kubelet/pods/378e604c-6be5-487a-a55a-669b6c22f9aa/volumes" Nov 02 20:16:56.183014 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:16:56.182999 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91074226-875a-4294-80f9-d761498e387a" path="/var/lib/kubelet/pods/91074226-875a-4294-80f9-d761498e387a/volumes" Nov 02 20:17:03.912556 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.912517 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb"] Nov 02 20:17:03.913025 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913007 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e530f339-18f4-44d0-85f2-33f984e993d3" containerName="revoke-keys" Nov 02 20:17:03.913025 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913020 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e530f339-18f4-44d0-85f2-33f984e993d3" containerName="revoke-keys" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913037 4966 cpu_manager.go:414] "containerMap: removing container" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913042 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913054 4966 cpu_manager.go:414] "containerMap: removing container" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913060 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913086 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d7499b11-1af4-4ca5-acef-68a5d6dcde9b" containerName="payload-processing" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913091 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7499b11-1af4-4ca5-acef-68a5d6dcde9b" containerName="payload-processing" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913161 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7499b11-1af4-4ca5-acef-68a5d6dcde9b" containerName="payload-processing" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913169 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e530f339-18f4-44d0-85f2-33f984e993d3" containerName="revoke-keys" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913177 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="91074226-875a-4294-80f9-d761498e387a" containerName="main" Nov 02 20:17:03.913181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.913184 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="378e604c-6be5-487a-a55a-669b6c22f9aa" containerName="main" Nov 02 20:17:03.918125 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.918106 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:03.962614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.962585 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4"] Nov 02 20:17:03.966463 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:03.966442 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.005345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.005308 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:17:04.005345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.005309 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:17:04.008625 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.008602 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb"] Nov 02 20:17:04.012441 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.012420 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:17:04.012558 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.012540 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:17:04.020763 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.020657 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.021084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.020837 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.021084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.020879 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.021084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.020949 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbsd\" (UniqueName: \"kubernetes.io/projected/e55edd7b-db3a-40b9-b23c-157349b3cc36-kube-api-access-9nbsd\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.021084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.020980 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsg4\" (UniqueName: \"kubernetes.io/projected/5790175d-42f0-4e57-abcd-fb125f7b359c-kube-api-access-sbsg4\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.021084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.021005 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.021482 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.021168 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4"] Nov 02 20:17:04.121798 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.121749 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.121798 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.121802 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.122055 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.121845 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbsd\" (UniqueName: \"kubernetes.io/projected/e55edd7b-db3a-40b9-b23c-157349b3cc36-kube-api-access-9nbsd\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.122055 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.121863 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsg4\" (UniqueName: \"kubernetes.io/projected/5790175d-42f0-4e57-abcd-fb125f7b359c-kube-api-access-sbsg4\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.122055 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.121881 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.122055 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.121928 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.122688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.122653 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.122688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.122686 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.124432 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.124410 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.124554 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.124413 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.174799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.174723 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbsd\" (UniqueName: \"kubernetes.io/projected/e55edd7b-db3a-40b9-b23c-157349b3cc36-kube-api-access-9nbsd\") pod \"maas-api-revoke-sub-1e2070a4-nq5lb\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.174799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.174778 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsg4\" (UniqueName: \"kubernetes.io/projected/5790175d-42f0-4e57-abcd-fb125f7b359c-kube-api-access-sbsg4\") pod \"maas-api-revoke-sub-90a5ec09-rmtr4\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.231429 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.231398 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:04.284092 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.283817 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:04.600958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.600926 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb"] Nov 02 20:17:04.601579 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:17:04.601555 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55edd7b_db3a_40b9_b23c_157349b3cc36.slice/crio-163c26aabb50f7379213ea4c09e80630e9f261a77be71e1d6670d2ef32685401 WatchSource:0}: Error finding container 163c26aabb50f7379213ea4c09e80630e9f261a77be71e1d6670d2ef32685401: Status 404 returned error can't find the container with id 163c26aabb50f7379213ea4c09e80630e9f261a77be71e1d6670d2ef32685401 Nov 02 20:17:04.693987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:04.693960 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4"] Nov 02 20:17:04.695342 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:17:04.695313 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5790175d_42f0_4e57_abcd_fb125f7b359c.slice/crio-8827e742d47379cfa9affdb02c92132728a228d0cee52497fc8c14587163cc17 WatchSource:0}: Error finding container 8827e742d47379cfa9affdb02c92132728a228d0cee52497fc8c14587163cc17: Status 404 returned error can't find the container with id 8827e742d47379cfa9affdb02c92132728a228d0cee52497fc8c14587163cc17 Nov 02 20:17:05.572573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:05.572534 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" event={"ID":"e55edd7b-db3a-40b9-b23c-157349b3cc36","Type":"ContainerStarted","Data":"7d4253b0285c2703035f13c6d403cda2a56ffd3294a3d9abe9e701df78cc750d"} Nov 02 20:17:05.573143 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:05.572593 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" event={"ID":"e55edd7b-db3a-40b9-b23c-157349b3cc36","Type":"ContainerStarted","Data":"163c26aabb50f7379213ea4c09e80630e9f261a77be71e1d6670d2ef32685401"} Nov 02 20:17:05.574602 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:05.574568 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" event={"ID":"5790175d-42f0-4e57-abcd-fb125f7b359c","Type":"ContainerStarted","Data":"0fb0a816663cea4054663df6871fb8225a8eaf3f51416c519434bcad0716569e"} Nov 02 20:17:05.574718 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:05.574626 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" event={"ID":"5790175d-42f0-4e57-abcd-fb125f7b359c","Type":"ContainerStarted","Data":"8827e742d47379cfa9affdb02c92132728a228d0cee52497fc8c14587163cc17"} Nov 02 20:17:05.680154 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:05.680079 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" podStartSLOduration=2.680059857 podStartE2EDuration="2.680059857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:05.677138516 +0000 UTC m=+2931.625794356" watchObservedRunningTime="2026-11-02 20:17:05.680059857 +0000 UTC m=+2931.628715696" Nov 02 20:17:05.786223 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:05.786164 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" podStartSLOduration=2.7861503499999998 podStartE2EDuration="2.78615035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:05.78348899 +0000 UTC m=+2931.732144829" watchObservedRunningTime="2026-11-02 20:17:05.78615035 +0000 UTC m=+2931.734806215" Nov 02 20:17:05.917867 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:05.917787 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" podUID="8754ec15-5b0b-4188-9f35-4d00a8b24043" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:17:06.587349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:06.587280 4966 generic.go:356] "Generic (PLEG): container finished" podID="e55edd7b-db3a-40b9-b23c-157349b3cc36" containerID="7d4253b0285c2703035f13c6d403cda2a56ffd3294a3d9abe9e701df78cc750d" exitCode=0 Nov 02 20:17:06.587921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:06.587361 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" event={"ID":"e55edd7b-db3a-40b9-b23c-157349b3cc36","Type":"ContainerDied","Data":"7d4253b0285c2703035f13c6d403cda2a56ffd3294a3d9abe9e701df78cc750d"} Nov 02 20:17:06.589872 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:06.589850 4966 generic.go:356] "Generic (PLEG): container finished" podID="5790175d-42f0-4e57-abcd-fb125f7b359c" containerID="0fb0a816663cea4054663df6871fb8225a8eaf3f51416c519434bcad0716569e" exitCode=0 Nov 02 20:17:06.589996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:06.589945 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" event={"ID":"5790175d-42f0-4e57-abcd-fb125f7b359c","Type":"ContainerDied","Data":"0fb0a816663cea4054663df6871fb8225a8eaf3f51416c519434bcad0716569e"} Nov 02 20:17:07.752987 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.752942 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:07.779795 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.779762 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.866850 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55edd7b-db3a-40b9-b23c-157349b3cc36-kube-api-access-9nbsd\" (UniqueName: \"kubernetes.io/projected/e55edd7b-db3a-40b9-b23c-157349b3cc36-kube-api-access-9nbsd\") pod \"e55edd7b-db3a-40b9-b23c-157349b3cc36\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.866909 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-xks-ca\") pod \"5790175d-42f0-4e57-abcd-fb125f7b359c\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.866944 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-openshift-ca\") pod \"e55edd7b-db3a-40b9-b23c-157349b3cc36\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.867022 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-openshift-ca\") pod \"5790175d-42f0-4e57-abcd-fb125f7b359c\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.867126 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-xks-ca\") pod \"e55edd7b-db3a-40b9-b23c-157349b3cc36\" (UID: \"e55edd7b-db3a-40b9-b23c-157349b3cc36\") " Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.867152 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5790175d-42f0-4e57-abcd-fb125f7b359c-kube-api-access-sbsg4\" (UniqueName: \"kubernetes.io/projected/5790175d-42f0-4e57-abcd-fb125f7b359c-kube-api-access-sbsg4\") pod \"5790175d-42f0-4e57-abcd-fb125f7b359c\" (UID: \"5790175d-42f0-4e57-abcd-fb125f7b359c\") " Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.867756 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-openshift-ca" pod "5790175d-42f0-4e57-abcd-fb125f7b359c" (UID: "5790175d-42f0-4e57-abcd-fb125f7b359c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:07.868744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.867926 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-openshift-ca" pod "e55edd7b-db3a-40b9-b23c-157349b3cc36" (UID: "e55edd7b-db3a-40b9-b23c-157349b3cc36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:07.870543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.870514 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5790175d-42f0-4e57-abcd-fb125f7b359c-kube-api-access-sbsg4" pod "5790175d-42f0-4e57-abcd-fb125f7b359c" (UID: "5790175d-42f0-4e57-abcd-fb125f7b359c"). InnerVolumeSpecName "kube-api-access-sbsg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:07.870814 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.870788 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-xks-ca" pod "e55edd7b-db3a-40b9-b23c-157349b3cc36" (UID: "e55edd7b-db3a-40b9-b23c-157349b3cc36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:07.872249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.872212 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-xks-ca" pod "5790175d-42f0-4e57-abcd-fb125f7b359c" (UID: "5790175d-42f0-4e57-abcd-fb125f7b359c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:07.887728 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.887700 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55edd7b-db3a-40b9-b23c-157349b3cc36-kube-api-access-9nbsd" pod "e55edd7b-db3a-40b9-b23c-157349b3cc36" (UID: "e55edd7b-db3a-40b9-b23c-157349b3cc36"). InnerVolumeSpecName "kube-api-access-9nbsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:07.968361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.968323 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nbsd\" (UniqueName: \"kubernetes.io/projected/e55edd7b-db3a-40b9-b23c-157349b3cc36-kube-api-access-9nbsd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:07.968361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.968357 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:07.968361 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.968368 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:07.968590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.968377 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5790175d-42f0-4e57-abcd-fb125f7b359c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:07.968590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.968386 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55edd7b-db3a-40b9-b23c-157349b3cc36-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:07.968590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:07.968394 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbsg4\" (UniqueName: \"kubernetes.io/projected/5790175d-42f0-4e57-abcd-fb125f7b359c-kube-api-access-sbsg4\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:08.536624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.536582 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:17:08.606083 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.606041 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" Nov 02 20:17:08.606083 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.606061 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb" event={"ID":"e55edd7b-db3a-40b9-b23c-157349b3cc36","Type":"ContainerDied","Data":"163c26aabb50f7379213ea4c09e80630e9f261a77be71e1d6670d2ef32685401"} Nov 02 20:17:08.606396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.606117 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="163c26aabb50f7379213ea4c09e80630e9f261a77be71e1d6670d2ef32685401" Nov 02 20:17:08.607853 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.607815 4966 generic.go:356] "Generic (PLEG): container finished" podID="8754ec15-5b0b-4188-9f35-4d00a8b24043" containerID="57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc" exitCode=0 Nov 02 20:17:08.607993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.607856 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" Nov 02 20:17:08.607993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.607891 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" event={"ID":"8754ec15-5b0b-4188-9f35-4d00a8b24043","Type":"ContainerDied","Data":"57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc"} Nov 02 20:17:08.607993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.607951 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs" event={"ID":"8754ec15-5b0b-4188-9f35-4d00a8b24043","Type":"ContainerDied","Data":"b1e71fdf39fff96ef7129ab58f4e37c3903ade85b0e44d9da05661ef9b013880"} Nov 02 20:17:08.607993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.607975 4966 scope.go:122] "RemoveContainer" containerID="57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc" Nov 02 20:17:08.609993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.609963 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" Nov 02 20:17:08.610094 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.610018 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4" event={"ID":"5790175d-42f0-4e57-abcd-fb125f7b359c","Type":"ContainerDied","Data":"8827e742d47379cfa9affdb02c92132728a228d0cee52497fc8c14587163cc17"} Nov 02 20:17:08.610094 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.610068 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8827e742d47379cfa9affdb02c92132728a228d0cee52497fc8c14587163cc17" Nov 02 20:17:08.620238 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.620211 4966 scope.go:122] "RemoveContainer" containerID="57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc" Nov 02 20:17:08.620558 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:17:08.620531 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc\": container with ID starting with 57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc not found: ID does not exist" containerID="57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc" Nov 02 20:17:08.620635 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.620572 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc"} err="failed to get container status \"57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc\": rpc error: code = NotFound desc = could not find container \"57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc\": container with ID starting with 57fdc944f0ee75a4e81b936418f8594de0a3b5ab38220fb255cea387994631cc not found: ID does not exist" Nov 02 20:17:08.678120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.678084 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8754ec15-5b0b-4188-9f35-4d00a8b24043-kube-api-access-x2jv9\" (UniqueName: \"kubernetes.io/projected/8754ec15-5b0b-4188-9f35-4d00a8b24043-kube-api-access-x2jv9\") pod \"8754ec15-5b0b-4188-9f35-4d00a8b24043\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " Nov 02 20:17:08.678351 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.678137 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8754ec15-5b0b-4188-9f35-4d00a8b24043-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8754ec15-5b0b-4188-9f35-4d00a8b24043-plugins-config-volume\") pod \"8754ec15-5b0b-4188-9f35-4d00a8b24043\" (UID: \"8754ec15-5b0b-4188-9f35-4d00a8b24043\") " Nov 02 20:17:08.678657 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.678625 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8754ec15-5b0b-4188-9f35-4d00a8b24043-plugins-config-volume" pod "8754ec15-5b0b-4188-9f35-4d00a8b24043" (UID: "8754ec15-5b0b-4188-9f35-4d00a8b24043"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:08.680483 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.680449 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8754ec15-5b0b-4188-9f35-4d00a8b24043-kube-api-access-x2jv9" pod "8754ec15-5b0b-4188-9f35-4d00a8b24043" (UID: "8754ec15-5b0b-4188-9f35-4d00a8b24043"). InnerVolumeSpecName "kube-api-access-x2jv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:08.779850 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.779816 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jv9\" (UniqueName: \"kubernetes.io/projected/8754ec15-5b0b-4188-9f35-4d00a8b24043-kube-api-access-x2jv9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:08.779850 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:08.779848 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8754ec15-5b0b-4188-9f35-4d00a8b24043-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:09.022108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:09.022027 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs"] Nov 02 20:17:09.025088 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:09.025062 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13e739-797694b9c5-82xfs"] Nov 02 20:17:10.184418 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:10.184381 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8754ec15-5b0b-4188-9f35-4d00a8b24043" path="/var/lib/kubelet/pods/8754ec15-5b0b-4188-9f35-4d00a8b24043/volumes" Nov 02 20:17:10.392039 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:10.391988 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt"] Nov 02 20:17:10.399673 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:10.399645 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13e739-4b9nt"] Nov 02 20:17:12.183859 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:12.183820 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85da80c-7079-438e-9c07-520561b8b6c3" path="/var/lib/kubelet/pods/c85da80c-7079-438e-9c07-520561b8b6c3/volumes" Nov 02 20:17:14.752850 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:17:14.752817 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:17:16.083574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:16.083537 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8"] Nov 02 20:17:16.130657 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:16.130580 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-d1aaed-dn9v8"] Nov 02 20:17:16.183501 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:16.183463 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e530f339-18f4-44d0-85f2-33f984e993d3" path="/var/lib/kubelet/pods/e530f339-18f4-44d0-85f2-33f984e993d3/volumes" Nov 02 20:17:18.823931 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.823894 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522"] Nov 02 20:17:18.824371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824358 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8754ec15-5b0b-4188-9f35-4d00a8b24043" containerName="payload-pre-processing" Nov 02 20:17:18.824425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824373 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8754ec15-5b0b-4188-9f35-4d00a8b24043" containerName="payload-pre-processing" Nov 02 20:17:18.824425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824394 4966 cpu_manager.go:414] "containerMap: removing container" podUID="e55edd7b-db3a-40b9-b23c-157349b3cc36" containerName="revoke-keys" Nov 02 20:17:18.824425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824399 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55edd7b-db3a-40b9-b23c-157349b3cc36" containerName="revoke-keys" Nov 02 20:17:18.824425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824414 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5790175d-42f0-4e57-abcd-fb125f7b359c" containerName="revoke-keys" Nov 02 20:17:18.824425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824419 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5790175d-42f0-4e57-abcd-fb125f7b359c" containerName="revoke-keys" Nov 02 20:17:18.824643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824503 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8754ec15-5b0b-4188-9f35-4d00a8b24043" containerName="payload-pre-processing" Nov 02 20:17:18.824643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824515 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55edd7b-db3a-40b9-b23c-157349b3cc36" containerName="revoke-keys" Nov 02 20:17:18.824643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.824521 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5790175d-42f0-4e57-abcd-fb125f7b359c" containerName="revoke-keys" Nov 02 20:17:18.828087 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.828069 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:18.923342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.923278 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:17:18.923342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.923329 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:17:18.923342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.923278 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:17:18.923687 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.923418 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:17:18.926715 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.926683 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522"] Nov 02 20:17:18.984885 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.984845 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:18.985142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.984927 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nsvw\" (UniqueName: \"kubernetes.io/projected/523b1811-d535-4a1f-915b-08f4930f1577-kube-api-access-2nsvw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:18.985142 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:18.985007 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.086371 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.086249 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.086561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.086385 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsvw\" (UniqueName: \"kubernetes.io/projected/523b1811-d535-4a1f-915b-08f4930f1577-kube-api-access-2nsvw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.086561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.086428 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.087066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.087037 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.088876 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.088855 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.139798 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.139766 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nsvw\" (UniqueName: \"kubernetes.io/projected/523b1811-d535-4a1f-915b-08f4930f1577-kube-api-access-2nsvw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.142535 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.142511 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:19.230587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.230556 4966 scope.go:122] "RemoveContainer" containerID="72f71525dc64ae24f778164e75417c50134a771817fd2c1ad0b9202af6f17ef9" Nov 02 20:17:19.511016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.510985 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522"] Nov 02 20:17:19.512119 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:17:19.512094 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523b1811_d535_4a1f_915b_08f4930f1577.slice/crio-4903c55a203fb72d578e1aafcec520286380ddea2016877bf20e6c9590c24b5d WatchSource:0}: Error finding container 4903c55a203fb72d578e1aafcec520286380ddea2016877bf20e6c9590c24b5d: Status 404 returned error can't find the container with id 4903c55a203fb72d578e1aafcec520286380ddea2016877bf20e6c9590c24b5d Nov 02 20:17:19.694486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:19.694449 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" event={"ID":"523b1811-d535-4a1f-915b-08f4930f1577","Type":"ContainerStarted","Data":"4903c55a203fb72d578e1aafcec520286380ddea2016877bf20e6c9590c24b5d"} Nov 02 20:17:20.702769 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:20.702729 4966 generic.go:356] "Generic (PLEG): container finished" podID="523b1811-d535-4a1f-915b-08f4930f1577" containerID="2d212ae271ce0fd81316261bbd44cde242b27a8dc30b7a58217ead036d788951" exitCode=0 Nov 02 20:17:20.703237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:20.702809 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" event={"ID":"523b1811-d535-4a1f-915b-08f4930f1577","Type":"ContainerDied","Data":"2d212ae271ce0fd81316261bbd44cde242b27a8dc30b7a58217ead036d788951"} Nov 02 20:17:21.879225 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:21.879186 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:22.018291 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.018255 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523b1811-d535-4a1f-915b-08f4930f1577-kube-api-access-2nsvw\" (UniqueName: \"kubernetes.io/projected/523b1811-d535-4a1f-915b-08f4930f1577-kube-api-access-2nsvw\") pod \"523b1811-d535-4a1f-915b-08f4930f1577\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " Nov 02 20:17:22.018494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.018369 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-xks-ca\") pod \"523b1811-d535-4a1f-915b-08f4930f1577\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " Nov 02 20:17:22.018494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.018437 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-openshift-ca\") pod \"523b1811-d535-4a1f-915b-08f4930f1577\" (UID: \"523b1811-d535-4a1f-915b-08f4930f1577\") " Nov 02 20:17:22.018981 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.018939 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-openshift-ca" pod "523b1811-d535-4a1f-915b-08f4930f1577" (UID: "523b1811-d535-4a1f-915b-08f4930f1577"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:22.020596 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.020568 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523b1811-d535-4a1f-915b-08f4930f1577-kube-api-access-2nsvw" pod "523b1811-d535-4a1f-915b-08f4930f1577" (UID: "523b1811-d535-4a1f-915b-08f4930f1577"). InnerVolumeSpecName "kube-api-access-2nsvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:22.021152 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.021131 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-xks-ca" pod "523b1811-d535-4a1f-915b-08f4930f1577" (UID: "523b1811-d535-4a1f-915b-08f4930f1577"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:22.120195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.120149 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:22.120195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.120188 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523b1811-d535-4a1f-915b-08f4930f1577-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:22.120195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.120200 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nsvw\" (UniqueName: \"kubernetes.io/projected/523b1811-d535-4a1f-915b-08f4930f1577-kube-api-access-2nsvw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:22.718590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.717379 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" Nov 02 20:17:22.718590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.717453 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522" event={"ID":"523b1811-d535-4a1f-915b-08f4930f1577","Type":"ContainerDied","Data":"4903c55a203fb72d578e1aafcec520286380ddea2016877bf20e6c9590c24b5d"} Nov 02 20:17:22.718590 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:22.717507 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4903c55a203fb72d578e1aafcec520286380ddea2016877bf20e6c9590c24b5d" Nov 02 20:17:23.576887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.576850 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g"] Nov 02 20:17:23.577586 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.577563 4966 cpu_manager.go:414] "containerMap: removing container" podUID="523b1811-d535-4a1f-915b-08f4930f1577" containerName="revoke-keys" Nov 02 20:17:23.577688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.577588 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523b1811-d535-4a1f-915b-08f4930f1577" containerName="revoke-keys" Nov 02 20:17:23.577756 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.577718 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="523b1811-d535-4a1f-915b-08f4930f1577" containerName="revoke-keys" Nov 02 20:17:23.581279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.581257 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.635083 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.635008 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.635386 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.635115 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.635386 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.635145 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rbx\" (UniqueName: \"kubernetes.io/projected/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-kube-api-access-29rbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.635578 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.635467 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g"] Nov 02 20:17:23.676489 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.676453 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:17:23.676749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.676622 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:17:23.676749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.676658 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:17:23.676749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.676666 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:17:23.736424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.736392 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.736638 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.736497 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.736638 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.736539 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29rbx\" (UniqueName: \"kubernetes.io/projected/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-kube-api-access-29rbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.737203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.737177 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.739345 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.739311 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.794857 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.794769 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rbx\" (UniqueName: \"kubernetes.io/projected/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-kube-api-access-29rbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:23.895467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:23.895390 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:24.278631 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:24.278599 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g"] Nov 02 20:17:24.279962 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:17:24.279921 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbf07f3_d41b_4fd0_b6ac_1e783139f0fc.slice/crio-c466cbffee06e85acc46990ab8e8358a74da746519c2fad6b40d5c98a7d1a58f WatchSource:0}: Error finding container c466cbffee06e85acc46990ab8e8358a74da746519c2fad6b40d5c98a7d1a58f: Status 404 returned error can't find the container with id c466cbffee06e85acc46990ab8e8358a74da746519c2fad6b40d5c98a7d1a58f Nov 02 20:17:24.734633 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:24.734593 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" event={"ID":"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc","Type":"ContainerStarted","Data":"0b724c2640367655d3450096c41e6a7946368c1378caa290334340377a5af5ea"} Nov 02 20:17:24.735174 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:24.734660 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" event={"ID":"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc","Type":"ContainerStarted","Data":"c466cbffee06e85acc46990ab8e8358a74da746519c2fad6b40d5c98a7d1a58f"} Nov 02 20:17:24.842505 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:24.842401 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" podStartSLOduration=1.8423861700000002 podStartE2EDuration="1.84238617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:24.839229991 +0000 UTC m=+2950.787885832" watchObservedRunningTime="2026-11-02 20:17:24.84238617 +0000 UTC m=+2950.791042008" Nov 02 20:17:25.743397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:25.743361 4966 generic.go:356] "Generic (PLEG): container finished" podID="7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" containerID="0b724c2640367655d3450096c41e6a7946368c1378caa290334340377a5af5ea" exitCode=0 Nov 02 20:17:25.743824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:25.743444 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" event={"ID":"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc","Type":"ContainerDied","Data":"0b724c2640367655d3450096c41e6a7946368c1378caa290334340377a5af5ea"} Nov 02 20:17:26.893798 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:26.893764 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:26.972778 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:26.972737 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-xks-ca\") pod \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " Nov 02 20:17:26.972991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:26.972844 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-kube-api-access-29rbx\" (UniqueName: \"kubernetes.io/projected/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-kube-api-access-29rbx\") pod \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " Nov 02 20:17:26.972991 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:26.972914 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-openshift-ca\") pod \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\" (UID: \"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc\") " Nov 02 20:17:26.973273 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:26.973248 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-openshift-ca" pod "7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" (UID: "7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:26.974990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:26.974971 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-xks-ca" pod "7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" (UID: "7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:26.975049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:26.975030 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-kube-api-access-29rbx" pod "7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" (UID: "7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc"). InnerVolumeSpecName "kube-api-access-29rbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:27.073812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:27.073728 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rbx\" (UniqueName: \"kubernetes.io/projected/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-kube-api-access-29rbx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:27.073812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:27.073760 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:27.073812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:27.073770 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:27.758348 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:27.758311 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" event={"ID":"7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc","Type":"ContainerDied","Data":"c466cbffee06e85acc46990ab8e8358a74da746519c2fad6b40d5c98a7d1a58f"} Nov 02 20:17:27.758547 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:27.758361 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g" Nov 02 20:17:27.758547 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:27.758365 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c466cbffee06e85acc46990ab8e8358a74da746519c2fad6b40d5c98a7d1a58f" Nov 02 20:17:35.118407 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:35.118367 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg"] Nov 02 20:17:35.234556 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:35.234522 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n"] Nov 02 20:17:40.118932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.118884 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" podUID="cd584daa-3872-4b5b-884c-d69457bdf6ab" containerName="payload-processing" containerID="cri-o://315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22" gracePeriod=25 Nov 02 20:17:40.235749 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.235708 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" podUID="ed90b751-2446-4d32-9e72-80255277cd17" containerName="payload-pre-processing" containerID="cri-o://e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745" gracePeriod=25 Nov 02 20:17:40.355234 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.355200 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz"] Nov 02 20:17:40.375858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.375787 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 20:17:40.401441 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.401407 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-43ceaa-mh5kz"] Nov 02 20:17:40.502152 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.502121 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 20:17:40.513692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.513662 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd584daa-3872-4b5b-884c-d69457bdf6ab-kube-api-access-48grq\" (UniqueName: \"kubernetes.io/projected/cd584daa-3872-4b5b-884c-d69457bdf6ab-kube-api-access-48grq\") pod \"cd584daa-3872-4b5b-884c-d69457bdf6ab\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " Nov 02 20:17:40.513897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.513806 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd584daa-3872-4b5b-884c-d69457bdf6ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd584daa-3872-4b5b-884c-d69457bdf6ab-plugins-config-volume\") pod \"cd584daa-3872-4b5b-884c-d69457bdf6ab\" (UID: \"cd584daa-3872-4b5b-884c-d69457bdf6ab\") " Nov 02 20:17:40.514220 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.514196 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd584daa-3872-4b5b-884c-d69457bdf6ab-plugins-config-volume" pod "cd584daa-3872-4b5b-884c-d69457bdf6ab" (UID: "cd584daa-3872-4b5b-884c-d69457bdf6ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:40.515801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.515770 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd584daa-3872-4b5b-884c-d69457bdf6ab-kube-api-access-48grq" pod "cd584daa-3872-4b5b-884c-d69457bdf6ab" (UID: "cd584daa-3872-4b5b-884c-d69457bdf6ab"). InnerVolumeSpecName "kube-api-access-48grq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:40.614532 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.614499 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed90b751-2446-4d32-9e72-80255277cd17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed90b751-2446-4d32-9e72-80255277cd17-plugins-config-volume\") pod \"ed90b751-2446-4d32-9e72-80255277cd17\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " Nov 02 20:17:40.614877 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.614626 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed90b751-2446-4d32-9e72-80255277cd17-kube-api-access-q8rgr\" (UniqueName: \"kubernetes.io/projected/ed90b751-2446-4d32-9e72-80255277cd17-kube-api-access-q8rgr\") pod \"ed90b751-2446-4d32-9e72-80255277cd17\" (UID: \"ed90b751-2446-4d32-9e72-80255277cd17\") " Nov 02 20:17:40.614990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.614902 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed90b751-2446-4d32-9e72-80255277cd17-plugins-config-volume" pod "ed90b751-2446-4d32-9e72-80255277cd17" (UID: "ed90b751-2446-4d32-9e72-80255277cd17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:40.614990 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.614982 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed90b751-2446-4d32-9e72-80255277cd17-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.615108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.615000 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd584daa-3872-4b5b-884c-d69457bdf6ab-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.615108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.615011 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48grq\" (UniqueName: \"kubernetes.io/projected/cd584daa-3872-4b5b-884c-d69457bdf6ab-kube-api-access-48grq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.616862 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.616829 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed90b751-2446-4d32-9e72-80255277cd17-kube-api-access-q8rgr" pod "ed90b751-2446-4d32-9e72-80255277cd17" (UID: "ed90b751-2446-4d32-9e72-80255277cd17"). InnerVolumeSpecName "kube-api-access-q8rgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:40.716565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.716489 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8rgr\" (UniqueName: \"kubernetes.io/projected/ed90b751-2446-4d32-9e72-80255277cd17-kube-api-access-q8rgr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.854748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.854709 4966 generic.go:356] "Generic (PLEG): container finished" podID="cd584daa-3872-4b5b-884c-d69457bdf6ab" containerID="315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22" exitCode=0 Nov 02 20:17:40.854935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.854761 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" Nov 02 20:17:40.854935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.854797 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" event={"ID":"cd584daa-3872-4b5b-884c-d69457bdf6ab","Type":"ContainerDied","Data":"315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22"} Nov 02 20:17:40.854935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.854854 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg" event={"ID":"cd584daa-3872-4b5b-884c-d69457bdf6ab","Type":"ContainerDied","Data":"b0643c86b9181218dd806edfcbd8226dddae4c9f5309b4586b7f914e9c986869"} Nov 02 20:17:40.854935 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.854876 4966 scope.go:122] "RemoveContainer" containerID="315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22" Nov 02 20:17:40.856216 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.856191 4966 generic.go:356] "Generic (PLEG): container finished" podID="ed90b751-2446-4d32-9e72-80255277cd17" containerID="e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745" exitCode=0 Nov 02 20:17:40.856382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.856237 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" Nov 02 20:17:40.856382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.856256 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" event={"ID":"ed90b751-2446-4d32-9e72-80255277cd17","Type":"ContainerDied","Data":"e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745"} Nov 02 20:17:40.856382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.856287 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n" event={"ID":"ed90b751-2446-4d32-9e72-80255277cd17","Type":"ContainerDied","Data":"c60ac48e0758c46098b49501ab4bae39ab3e154f6c4f1b6a0ae370b7a4480dac"} Nov 02 20:17:40.865765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.865748 4966 scope.go:122] "RemoveContainer" containerID="315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22" Nov 02 20:17:40.866006 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:17:40.865986 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22\": container with ID starting with 315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22 not found: ID does not exist" containerID="315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22" Nov 02 20:17:40.866103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.866017 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22"} err="failed to get container status \"315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22\": rpc error: code = NotFound desc = could not find container \"315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22\": container with ID starting with 315ad0eb0a8a37e155ae7676aa930cbe03580343fd15d642cbc5c52b60ddbf22 not found: ID does not exist" Nov 02 20:17:40.866103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.866037 4966 scope.go:122] "RemoveContainer" containerID="e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745" Nov 02 20:17:40.875503 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.875479 4966 scope.go:122] "RemoveContainer" containerID="e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745" Nov 02 20:17:40.875757 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:17:40.875742 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745\": container with ID starting with e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745 not found: ID does not exist" containerID="e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745" Nov 02 20:17:40.875808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:40.875763 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745"} err="failed to get container status \"e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745\": rpc error: code = NotFound desc = could not find container \"e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745\": container with ID starting with e20c542e1a01f82ff7ac9125946edf8b41dbf8b159f779a4355f24837998a745 not found: ID does not exist" Nov 02 20:17:41.007485 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:41.007451 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg"] Nov 02 20:17:41.053388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:41.053350 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24ef8b-5d88597b9-g2zhg"] Nov 02 20:17:41.153382 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:41.153347 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n"] Nov 02 20:17:41.156976 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:41.156947 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24ef8b-bdff7c8bc-7jk8n"] Nov 02 20:17:42.183181 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:42.183139 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64ef426-a7e2-4a6c-9ef1-655c960ded11" path="/var/lib/kubelet/pods/c64ef426-a7e2-4a6c-9ef1-655c960ded11/volumes" Nov 02 20:17:42.183751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:42.183729 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd584daa-3872-4b5b-884c-d69457bdf6ab" path="/var/lib/kubelet/pods/cd584daa-3872-4b5b-884c-d69457bdf6ab/volumes" Nov 02 20:17:42.184174 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:42.184155 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed90b751-2446-4d32-9e72-80255277cd17" path="/var/lib/kubelet/pods/ed90b751-2446-4d32-9e72-80255277cd17/volumes" Nov 02 20:17:44.753679 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:17:44.753643 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:17:45.988025 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:45.987978 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522"] Nov 02 20:17:46.029384 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:46.029351 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4848d2-nr522"] Nov 02 20:17:46.183744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:46.183697 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523b1811-d535-4a1f-915b-08f4930f1577" path="/var/lib/kubelet/pods/523b1811-d535-4a1f-915b-08f4930f1577/volumes" Nov 02 20:17:47.212634 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:47.212600 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5"] Nov 02 20:17:47.213058 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:47.212814 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" containerID="cri-o://523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7" gracePeriod=30 Nov 02 20:17:47.244100 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:47.244061 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g"] Nov 02 20:17:47.248552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:47.248518 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-24ef8b-92d4g"] Nov 02 20:17:48.182904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:48.182866 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" path="/var/lib/kubelet/pods/7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc/volumes" Nov 02 20:17:52.478712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.478684 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 20:17:52.642325 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642214 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-istio-token\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-istio-token\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.642325 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642316 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-credential-socket\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.642621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642397 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-certs\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.642621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642464 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-kube-api-access-ngc5b\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-kube-api-access-ngc5b\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.642621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642518 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-data\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.642621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642558 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fca23981-937a-43ec-8571-065f11cd2175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca23981-937a-43ec-8571-065f11cd2175-istio-podinfo\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.642621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642599 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-envoy\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.643866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642645 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-socket\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.643866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.642694 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca23981-937a-43ec-8571-065f11cd2175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca23981-937a-43ec-8571-065f11cd2175-istiod-ca-cert\") pod \"fca23981-937a-43ec-8571-065f11cd2175\" (UID: \"fca23981-937a-43ec-8571-065f11cd2175\") " Nov 02 20:17:52.643866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.643336 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-certs" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:52.643866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.643500 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-data" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:52.643866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.643573 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca23981-937a-43ec-8571-065f11cd2175-istiod-ca-cert" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:52.643866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.643690 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-socket" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:52.643866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.643744 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-credential-socket" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:52.647117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.647074 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-istio-token" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:52.647490 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.647462 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fca23981-937a-43ec-8571-065f11cd2175-istio-podinfo" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:17:52.647588 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.647487 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-kube-api-access-ngc5b" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "kube-api-access-ngc5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:52.652850 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.652798 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-envoy" pod "fca23981-937a-43ec-8571-065f11cd2175" (UID: "fca23981-937a-43ec-8571-065f11cd2175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:52.744114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744081 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngc5b\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-kube-api-access-ngc5b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744108 4966 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744118 4966 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca23981-937a-43ec-8571-065f11cd2175-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744129 4966 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744138 4966 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744147 4966 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca23981-937a-43ec-8571-065f11cd2175-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744154 4966 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca23981-937a-43ec-8571-065f11cd2175-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744162 4966 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.744359 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.744171 4966 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca23981-937a-43ec-8571-065f11cd2175-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:52.945844 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.945748 4966 generic.go:356] "Generic (PLEG): container finished" podID="fca23981-937a-43ec-8571-065f11cd2175" containerID="523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7" exitCode=0 Nov 02 20:17:52.945844 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.945824 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" Nov 02 20:17:52.946115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.945839 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" event={"ID":"fca23981-937a-43ec-8571-065f11cd2175","Type":"ContainerDied","Data":"523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7"} Nov 02 20:17:52.946115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.945906 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5" event={"ID":"fca23981-937a-43ec-8571-065f11cd2175","Type":"ContainerDied","Data":"013d235dd5a570e01c483230f8935fc00d574eb778ae9edb2fc4ac1903c14505"} Nov 02 20:17:52.946115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.945937 4966 scope.go:122] "RemoveContainer" containerID="523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7" Nov 02 20:17:52.957670 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.957649 4966 scope.go:122] "RemoveContainer" containerID="523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7" Nov 02 20:17:52.958013 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:17:52.957985 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7\": container with ID starting with 523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7 not found: ID does not exist" containerID="523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7" Nov 02 20:17:52.958074 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:52.958031 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7"} err="failed to get container status \"523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7\": rpc error: code = NotFound desc = could not find container \"523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7\": container with ID starting with 523e2ed125227e8b005564d74ae39d75c1b4e61854452e81015b77472b7636b7 not found: ID does not exist" Nov 02 20:17:53.098524 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.098486 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5"] Nov 02 20:17:53.144675 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.144636 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-43ceaa-openshift-default-6c4c98d4b7-f9ts5"] Nov 02 20:17:53.301486 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.301451 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr"] Nov 02 20:17:53.301995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.301974 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ed90b751-2446-4d32-9e72-80255277cd17" containerName="payload-pre-processing" Nov 02 20:17:53.302071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.301997 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed90b751-2446-4d32-9e72-80255277cd17" containerName="payload-pre-processing" Nov 02 20:17:53.302071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302052 4966 cpu_manager.go:414] "containerMap: removing container" podUID="7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" containerName="revoke-keys" Nov 02 20:17:53.302071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302060 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" containerName="revoke-keys" Nov 02 20:17:53.302188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302074 4966 cpu_manager.go:414] "containerMap: removing container" podUID="cd584daa-3872-4b5b-884c-d69457bdf6ab" containerName="payload-processing" Nov 02 20:17:53.302188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302082 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd584daa-3872-4b5b-884c-d69457bdf6ab" containerName="payload-processing" Nov 02 20:17:53.302188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302101 4966 cpu_manager.go:414] "containerMap: removing container" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" Nov 02 20:17:53.302188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302109 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" Nov 02 20:17:53.302383 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302244 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbf07f3-d41b-4fd0-b6ac-1e783139f0fc" containerName="revoke-keys" Nov 02 20:17:53.302383 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302256 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd584daa-3872-4b5b-884c-d69457bdf6ab" containerName="payload-processing" Nov 02 20:17:53.302383 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302271 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca23981-937a-43ec-8571-065f11cd2175" containerName="istio-proxy" Nov 02 20:17:53.302383 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.302281 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed90b751-2446-4d32-9e72-80255277cd17" containerName="payload-pre-processing" Nov 02 20:17:53.306618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.306595 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.355284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.355248 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:17:53.355523 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.355502 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:17:53.355621 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.355585 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:17:53.355731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.355713 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:17:53.359645 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.359620 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr"] Nov 02 20:17:53.451720 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.451684 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.451720 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.451724 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzq5\" (UniqueName: \"kubernetes.io/projected/24652ba6-4d94-40f4-b85a-f00adad4183e-kube-api-access-5zzq5\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.451980 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.451889 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.552746 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.552656 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.552746 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.552713 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.552746 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.552735 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzq5\" (UniqueName: \"kubernetes.io/projected/24652ba6-4d94-40f4-b85a-f00adad4183e-kube-api-access-5zzq5\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.553362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.553342 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.555138 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.555113 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.605741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.605703 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzq5\" (UniqueName: \"kubernetes.io/projected/24652ba6-4d94-40f4-b85a-f00adad4183e-kube-api-access-5zzq5\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.621705 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.621665 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:53.995413 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:53.995382 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr"] Nov 02 20:17:53.996486 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:17:53.996461 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24652ba6_4d94_40f4_b85a_f00adad4183e.slice/crio-fd3057d8263ee5283d947d57e6768d03645486c0e9707714924fff7f358bbf50 WatchSource:0}: Error finding container fd3057d8263ee5283d947d57e6768d03645486c0e9707714924fff7f358bbf50: Status 404 returned error can't find the container with id fd3057d8263ee5283d947d57e6768d03645486c0e9707714924fff7f358bbf50 Nov 02 20:17:54.183423 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.183385 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca23981-937a-43ec-8571-065f11cd2175" path="/var/lib/kubelet/pods/fca23981-937a-43ec-8571-065f11cd2175/volumes" Nov 02 20:17:54.769001 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.768965 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p"] Nov 02 20:17:54.772888 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.772870 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.827227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.827188 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p"] Nov 02 20:17:54.865099 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.865058 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvq5\" (UniqueName: \"kubernetes.io/projected/45ce86f7-8400-45ce-b4cb-346fa98358f5-kube-api-access-7bvq5\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.865276 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.865103 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.865276 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.865212 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.963372 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.963338 4966 generic.go:356] "Generic (PLEG): container finished" podID="24652ba6-4d94-40f4-b85a-f00adad4183e" containerID="e526de837f87e10d8f52ca3078c4e62108a50d3fa782b327ee698b35a2ebf243" exitCode=0 Nov 02 20:17:54.963563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.963430 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" event={"ID":"24652ba6-4d94-40f4-b85a-f00adad4183e","Type":"ContainerDied","Data":"e526de837f87e10d8f52ca3078c4e62108a50d3fa782b327ee698b35a2ebf243"} Nov 02 20:17:54.963563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.963473 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" event={"ID":"24652ba6-4d94-40f4-b85a-f00adad4183e","Type":"ContainerStarted","Data":"fd3057d8263ee5283d947d57e6768d03645486c0e9707714924fff7f358bbf50"} Nov 02 20:17:54.965996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.965975 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.966089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.966076 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvq5\" (UniqueName: \"kubernetes.io/projected/45ce86f7-8400-45ce-b4cb-346fa98358f5-kube-api-access-7bvq5\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.966157 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.966099 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.966835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.966807 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:54.968563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:54.968540 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:55.019802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:55.019721 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvq5\" (UniqueName: \"kubernetes.io/projected/45ce86f7-8400-45ce-b4cb-346fa98358f5-kube-api-access-7bvq5\") pod \"maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:55.086411 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:55.086379 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:55.454743 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:55.454709 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p"] Nov 02 20:17:55.455856 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:17:55.455826 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ce86f7_8400_45ce_b4cb_346fa98358f5.slice/crio-03da8f1f0a1bb872e2088e8aca24cee6168f31d192b399799aa68f64527a7658 WatchSource:0}: Error finding container 03da8f1f0a1bb872e2088e8aca24cee6168f31d192b399799aa68f64527a7658: Status 404 returned error can't find the container with id 03da8f1f0a1bb872e2088e8aca24cee6168f31d192b399799aa68f64527a7658 Nov 02 20:17:55.972255 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:55.972210 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" event={"ID":"45ce86f7-8400-45ce-b4cb-346fa98358f5","Type":"ContainerStarted","Data":"f6be1e8533fb66114f6e47a55337997f79da4672beb3f2b685e3a83852cfe704"} Nov 02 20:17:55.972766 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:55.972313 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" event={"ID":"45ce86f7-8400-45ce-b4cb-346fa98358f5","Type":"ContainerStarted","Data":"03da8f1f0a1bb872e2088e8aca24cee6168f31d192b399799aa68f64527a7658"} Nov 02 20:17:56.128108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.128052 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" podStartSLOduration=2.128033003 podStartE2EDuration="2.128033003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:56.081119067 +0000 UTC m=+2982.029774899" watchObservedRunningTime="2026-11-02 20:17:56.128033003 +0000 UTC m=+2982.076688836" Nov 02 20:17:56.136608 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.136578 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:56.278103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.278065 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-xks-ca\") pod \"24652ba6-4d94-40f4-b85a-f00adad4183e\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " Nov 02 20:17:56.278343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.278167 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-openshift-ca\") pod \"24652ba6-4d94-40f4-b85a-f00adad4183e\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " Nov 02 20:17:56.278343 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.278195 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24652ba6-4d94-40f4-b85a-f00adad4183e-kube-api-access-5zzq5\" (UniqueName: \"kubernetes.io/projected/24652ba6-4d94-40f4-b85a-f00adad4183e-kube-api-access-5zzq5\") pod \"24652ba6-4d94-40f4-b85a-f00adad4183e\" (UID: \"24652ba6-4d94-40f4-b85a-f00adad4183e\") " Nov 02 20:17:56.278611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.278588 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-openshift-ca" pod "24652ba6-4d94-40f4-b85a-f00adad4183e" (UID: "24652ba6-4d94-40f4-b85a-f00adad4183e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:56.280502 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.280437 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24652ba6-4d94-40f4-b85a-f00adad4183e-kube-api-access-5zzq5" pod "24652ba6-4d94-40f4-b85a-f00adad4183e" (UID: "24652ba6-4d94-40f4-b85a-f00adad4183e"). InnerVolumeSpecName "kube-api-access-5zzq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:56.280502 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.280453 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-xks-ca" pod "24652ba6-4d94-40f4-b85a-f00adad4183e" (UID: "24652ba6-4d94-40f4-b85a-f00adad4183e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:56.379752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.379714 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:56.379752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.379751 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzq5\" (UniqueName: \"kubernetes.io/projected/24652ba6-4d94-40f4-b85a-f00adad4183e-kube-api-access-5zzq5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:56.380032 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.379769 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24652ba6-4d94-40f4-b85a-f00adad4183e-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:56.980175 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.980146 4966 generic.go:356] "Generic (PLEG): container finished" podID="45ce86f7-8400-45ce-b4cb-346fa98358f5" containerID="f6be1e8533fb66114f6e47a55337997f79da4672beb3f2b685e3a83852cfe704" exitCode=0 Nov 02 20:17:56.980647 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.980237 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" event={"ID":"45ce86f7-8400-45ce-b4cb-346fa98358f5","Type":"ContainerDied","Data":"f6be1e8533fb66114f6e47a55337997f79da4672beb3f2b685e3a83852cfe704"} Nov 02 20:17:56.981676 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.981654 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" Nov 02 20:17:56.981733 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.981698 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr" event={"ID":"24652ba6-4d94-40f4-b85a-f00adad4183e","Type":"ContainerDied","Data":"fd3057d8263ee5283d947d57e6768d03645486c0e9707714924fff7f358bbf50"} Nov 02 20:17:56.981782 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:56.981736 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd3057d8263ee5283d947d57e6768d03645486c0e9707714924fff7f358bbf50" Nov 02 20:17:58.146889 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.146856 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:58.296710 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.296572 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ce86f7-8400-45ce-b4cb-346fa98358f5-kube-api-access-7bvq5\" (UniqueName: \"kubernetes.io/projected/45ce86f7-8400-45ce-b4cb-346fa98358f5-kube-api-access-7bvq5\") pod \"45ce86f7-8400-45ce-b4cb-346fa98358f5\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " Nov 02 20:17:58.296710 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.296702 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-xks-ca\") pod \"45ce86f7-8400-45ce-b4cb-346fa98358f5\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " Nov 02 20:17:58.296921 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.296791 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-openshift-ca\") pod \"45ce86f7-8400-45ce-b4cb-346fa98358f5\" (UID: \"45ce86f7-8400-45ce-b4cb-346fa98358f5\") " Nov 02 20:17:58.297207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.297162 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-openshift-ca" pod "45ce86f7-8400-45ce-b4cb-346fa98358f5" (UID: "45ce86f7-8400-45ce-b4cb-346fa98358f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:58.297387 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.297348 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.299317 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.299280 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-xks-ca" pod "45ce86f7-8400-45ce-b4cb-346fa98358f5" (UID: "45ce86f7-8400-45ce-b4cb-346fa98358f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:58.299408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.299337 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ce86f7-8400-45ce-b4cb-346fa98358f5-kube-api-access-7bvq5" pod "45ce86f7-8400-45ce-b4cb-346fa98358f5" (UID: "45ce86f7-8400-45ce-b4cb-346fa98358f5"). InnerVolumeSpecName "kube-api-access-7bvq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:58.398244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.398210 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bvq5\" (UniqueName: \"kubernetes.io/projected/45ce86f7-8400-45ce-b4cb-346fa98358f5-kube-api-access-7bvq5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.398244 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.398243 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ce86f7-8400-45ce-b4cb-346fa98358f5-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.996867 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.996836 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" Nov 02 20:17:58.996867 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.996861 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p" event={"ID":"45ce86f7-8400-45ce-b4cb-346fa98358f5","Type":"ContainerDied","Data":"03da8f1f0a1bb872e2088e8aca24cee6168f31d192b399799aa68f64527a7658"} Nov 02 20:17:58.997085 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:58.996910 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03da8f1f0a1bb872e2088e8aca24cee6168f31d192b399799aa68f64527a7658" Nov 02 20:17:59.277501 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:59.277422 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz"] Nov 02 20:17:59.283932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:59.283894 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k"] Nov 02 20:17:59.294526 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:17:59.294496 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm"] Nov 02 20:18:07.451601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:07.451559 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49"] Nov 02 20:18:07.567033 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:07.566999 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv"] Nov 02 20:18:10.801655 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:10.801615 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr"] Nov 02 20:18:10.847467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:10.847435 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-282875-2nkxr"] Nov 02 20:18:12.182364 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:12.182326 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24652ba6-4d94-40f4-b85a-f00adad4183e" path="/var/lib/kubelet/pods/24652ba6-4d94-40f4-b85a-f00adad4183e/volumes" Nov 02 20:18:12.452334 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:12.452192 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" podUID="3ace283e-959e-4523-b74b-680ad65469bf" containerName="payload-processing" containerID="cri-o://b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736" gracePeriod=25 Nov 02 20:18:12.568321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:12.568242 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" podUID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" containerName="payload-pre-processing" containerID="cri-o://410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422" gracePeriod=25 Nov 02 20:18:14.302064 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.302012 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" containerID="cri-o://64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2" gracePeriod=45 Nov 02 20:18:14.307102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.307066 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8000/ready\": EOF" Nov 02 20:18:14.313842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.313803 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" containerID="cri-o://66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e" gracePeriod=45 Nov 02 20:18:14.319809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.319743 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" containerID="cri-o://def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741" gracePeriod=45 Nov 02 20:18:14.373407 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.373375 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused" Nov 02 20:18:14.600687 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.600653 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 20:18:14.637191 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.637167 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 20:18:14.644666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.644641 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 20:18:14.665665 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665622 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-model-cache\") pod \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " Nov 02 20:18:14.665843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665690 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-home\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-home\") pod \"772633d4-a7f3-47f3-a51e-fc351e9e366c\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " Nov 02 20:18:14.665843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665740 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-model-cache\") pod \"772633d4-a7f3-47f3-a51e-fc351e9e366c\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " Nov 02 20:18:14.665843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665778 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tmp-dir\") pod \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " Nov 02 20:18:14.665843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665794 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-dshm\") pod \"772633d4-a7f3-47f3-a51e-fc351e9e366c\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " Nov 02 20:18:14.665843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665823 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tls-certs\") pod \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " Nov 02 20:18:14.665843 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665844 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0587f2-6194-414c-a41a-ac1d9669dc0d-kube-api-access-r6w4g\" (UniqueName: \"kubernetes.io/projected/5d0587f2-6194-414c-a41a-ac1d9669dc0d-kube-api-access-r6w4g\") pod \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665865 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-dshm\") pod \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665885 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772633d4-a7f3-47f3-a51e-fc351e9e366c-kube-api-access-dmnnk\" (UniqueName: \"kubernetes.io/projected/772633d4-a7f3-47f3-a51e-fc351e9e366c-kube-api-access-dmnnk\") pod \"772633d4-a7f3-47f3-a51e-fc351e9e366c\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665894 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-model-cache" pod "6cb81dd2-012d-4a42-9090-5e204dd5f18a" (UID: "6cb81dd2-012d-4a42-9090-5e204dd5f18a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665910 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-dshm\") pod \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665929 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb81dd2-012d-4a42-9090-5e204dd5f18a-kube-api-access-wj4dw\" (UniqueName: \"kubernetes.io/projected/6cb81dd2-012d-4a42-9090-5e204dd5f18a-kube-api-access-wj4dw\") pod \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665961 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-home\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-home\") pod \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.665984 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-model-cache\") pod \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666021 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tls-certs\") pod \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666039 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-home\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-home\") pod \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\" (UID: \"6cb81dd2-012d-4a42-9090-5e204dd5f18a\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666063 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-tmp-dir\") pod \"772633d4-a7f3-47f3-a51e-fc351e9e366c\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666087 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tmp-dir\") pod \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\" (UID: \"5d0587f2-6194-414c-a41a-ac1d9669dc0d\") " Nov 02 20:18:14.666207 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666127 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772633d4-a7f3-47f3-a51e-fc351e9e366c-tls-certs\" (UniqueName: \"kubernetes.io/secret/772633d4-a7f3-47f3-a51e-fc351e9e366c-tls-certs\") pod \"772633d4-a7f3-47f3-a51e-fc351e9e366c\" (UID: \"772633d4-a7f3-47f3-a51e-fc351e9e366c\") " Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666309 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tmp-dir" pod "6cb81dd2-012d-4a42-9090-5e204dd5f18a" (UID: "6cb81dd2-012d-4a42-9090-5e204dd5f18a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666479 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-model-cache" pod "772633d4-a7f3-47f3-a51e-fc351e9e366c" (UID: "772633d4-a7f3-47f3-a51e-fc351e9e366c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666528 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666549 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666738 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-home" pod "772633d4-a7f3-47f3-a51e-fc351e9e366c" (UID: "772633d4-a7f3-47f3-a51e-fc351e9e366c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.666898 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-model-cache" pod "5d0587f2-6194-414c-a41a-ac1d9669dc0d" (UID: "5d0587f2-6194-414c-a41a-ac1d9669dc0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.667177 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-home" pod "6cb81dd2-012d-4a42-9090-5e204dd5f18a" (UID: "6cb81dd2-012d-4a42-9090-5e204dd5f18a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.667388 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-tmp-dir" pod "772633d4-a7f3-47f3-a51e-fc351e9e366c" (UID: "772633d4-a7f3-47f3-a51e-fc351e9e366c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.667431 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-home" pod "5d0587f2-6194-414c-a41a-ac1d9669dc0d" (UID: "5d0587f2-6194-414c-a41a-ac1d9669dc0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.667781 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.667583 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tmp-dir" pod "5d0587f2-6194-414c-a41a-ac1d9669dc0d" (UID: "5d0587f2-6194-414c-a41a-ac1d9669dc0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.668679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.668643 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tls-certs" pod "6cb81dd2-012d-4a42-9090-5e204dd5f18a" (UID: "6cb81dd2-012d-4a42-9090-5e204dd5f18a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:14.669447 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.669424 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-dshm" pod "772633d4-a7f3-47f3-a51e-fc351e9e366c" (UID: "772633d4-a7f3-47f3-a51e-fc351e9e366c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.669567 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.669529 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772633d4-a7f3-47f3-a51e-fc351e9e366c-tls-certs" pod "772633d4-a7f3-47f3-a51e-fc351e9e366c" (UID: "772633d4-a7f3-47f3-a51e-fc351e9e366c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:14.669638 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.669614 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772633d4-a7f3-47f3-a51e-fc351e9e366c-kube-api-access-dmnnk" pod "772633d4-a7f3-47f3-a51e-fc351e9e366c" (UID: "772633d4-a7f3-47f3-a51e-fc351e9e366c"). InnerVolumeSpecName "kube-api-access-dmnnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:14.670603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.670528 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-dshm" pod "6cb81dd2-012d-4a42-9090-5e204dd5f18a" (UID: "6cb81dd2-012d-4a42-9090-5e204dd5f18a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.671147 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.671123 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tls-certs" pod "5d0587f2-6194-414c-a41a-ac1d9669dc0d" (UID: "5d0587f2-6194-414c-a41a-ac1d9669dc0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:14.671256 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.671207 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-dshm" pod "5d0587f2-6194-414c-a41a-ac1d9669dc0d" (UID: "5d0587f2-6194-414c-a41a-ac1d9669dc0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:14.671336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.671293 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0587f2-6194-414c-a41a-ac1d9669dc0d-kube-api-access-r6w4g" pod "5d0587f2-6194-414c-a41a-ac1d9669dc0d" (UID: "5d0587f2-6194-414c-a41a-ac1d9669dc0d"). InnerVolumeSpecName "kube-api-access-r6w4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:14.671739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.671717 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb81dd2-012d-4a42-9090-5e204dd5f18a-kube-api-access-wj4dw" pod "6cb81dd2-012d-4a42-9090-5e204dd5f18a" (UID: "6cb81dd2-012d-4a42-9090-5e204dd5f18a"). InnerVolumeSpecName "kube-api-access-wj4dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:14.760934 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:14.760897 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:18:14.767642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767611 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767638 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767648 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767656 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767664 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772633d4-a7f3-47f3-a51e-fc351e9e366c-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767673 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767687 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767699 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772633d4-a7f3-47f3-a51e-fc351e9e366c-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767707 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb81dd2-012d-4a42-9090-5e204dd5f18a-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767715 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6w4g\" (UniqueName: \"kubernetes.io/projected/5d0587f2-6194-414c-a41a-ac1d9669dc0d-kube-api-access-r6w4g\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767723 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb81dd2-012d-4a42-9090-5e204dd5f18a-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767731 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnnk\" (UniqueName: \"kubernetes.io/projected/772633d4-a7f3-47f3-a51e-fc351e9e366c-kube-api-access-dmnnk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767739 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767748 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4dw\" (UniqueName: \"kubernetes.io/projected/6cb81dd2-012d-4a42-9090-5e204dd5f18a-kube-api-access-wj4dw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767761 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:14.767892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:14.767774 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0587f2-6194-414c-a41a-ac1d9669dc0d-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:15.099741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.099708 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:18:15.114632 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.114605 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:18:15.115179 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.115141 4966 generic.go:356] "Generic (PLEG): container finished" podID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerID="def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741" exitCode=0 Nov 02 20:18:15.115388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.115219 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" Nov 02 20:18:15.115388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.115224 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" event={"ID":"6cb81dd2-012d-4a42-9090-5e204dd5f18a","Type":"ContainerDied","Data":"def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741"} Nov 02 20:18:15.115388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.115286 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm" event={"ID":"6cb81dd2-012d-4a42-9090-5e204dd5f18a","Type":"ContainerDied","Data":"a0f0ec8f7b88c4999ac506241bc22665f8a0e32b710bace2eb13c263d1608c26"} Nov 02 20:18:15.115388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.115324 4966 scope.go:122] "RemoveContainer" containerID="def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741" Nov 02 20:18:15.116798 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.116769 4966 generic.go:356] "Generic (PLEG): container finished" podID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerID="66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e" exitCode=0 Nov 02 20:18:15.116920 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.116834 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" Nov 02 20:18:15.116920 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.116893 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" event={"ID":"772633d4-a7f3-47f3-a51e-fc351e9e366c","Type":"ContainerDied","Data":"66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e"} Nov 02 20:18:15.117039 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.116928 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k" event={"ID":"772633d4-a7f3-47f3-a51e-fc351e9e366c","Type":"ContainerDied","Data":"99808b00d3c41c62a21bdf398c23234f384432851b561e4bb0ac68019103cf46"} Nov 02 20:18:15.118459 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.118441 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:18:15.118827 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.118799 4966 generic.go:356] "Generic (PLEG): container finished" podID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerID="64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2" exitCode=0 Nov 02 20:18:15.118938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.118860 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" Nov 02 20:18:15.118938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.118871 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" event={"ID":"5d0587f2-6194-414c-a41a-ac1d9669dc0d","Type":"ContainerDied","Data":"64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2"} Nov 02 20:18:15.118938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.118905 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz" event={"ID":"5d0587f2-6194-414c-a41a-ac1d9669dc0d","Type":"ContainerDied","Data":"5e2616c67eb87b61ed2ae0d82885184fc352ed0ba8571457671666b4dce68c06"} Nov 02 20:18:15.127952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.127918 4966 scope.go:122] "RemoveContainer" containerID="def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741" Nov 02 20:18:15.128257 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:15.128230 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741\": container with ID starting with def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741 not found: ID does not exist" containerID="def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741" Nov 02 20:18:15.128400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.128264 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741"} err="failed to get container status \"def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741\": rpc error: code = NotFound desc = could not find container \"def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741\": container with ID starting with def4f81a19e3d64905aaa3890c254e7058f10701ff57dc015ca774d79287b741 not found: ID does not exist" Nov 02 20:18:15.128400 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.128289 4966 scope.go:122] "RemoveContainer" containerID="66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e" Nov 02 20:18:15.140431 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.140412 4966 scope.go:122] "RemoveContainer" containerID="66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e" Nov 02 20:18:15.140594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.140571 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:18:15.140763 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:15.140741 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e\": container with ID starting with 66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e not found: ID does not exist" containerID="66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e" Nov 02 20:18:15.140907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.140772 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e"} err="failed to get container status \"66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e\": rpc error: code = NotFound desc = could not find container \"66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e\": container with ID starting with 66bf96cf6d004f5ea2b85a6e0bfe73b478677b7513251d60717422c56927d55e not found: ID does not exist" Nov 02 20:18:15.140907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.140794 4966 scope.go:122] "RemoveContainer" containerID="64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2" Nov 02 20:18:15.153408 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.153371 4966 scope.go:122] "RemoveContainer" containerID="64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2" Nov 02 20:18:15.153685 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:15.153664 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2\": container with ID starting with 64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2 not found: ID does not exist" containerID="64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2" Nov 02 20:18:15.153752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.153693 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2"} err="failed to get container status \"64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2\": rpc error: code = NotFound desc = could not find container \"64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2\": container with ID starting with 64f42a6b14a54c7db851439909278afedf05d694f5dcecf99b10a76e7be359a2 not found: ID does not exist" Nov 02 20:18:15.268469 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.268433 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k"] Nov 02 20:18:15.271203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.271173 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-w6-e9c144-kserve-564c795857-njq2k"] Nov 02 20:18:15.371353 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.371245 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm"] Nov 02 20:18:15.375640 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.375608 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/premium-w6-e9c144-kserve-677dc85f6b-2kdlm"] Nov 02 20:18:15.475385 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.475350 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz"] Nov 02 20:18:15.480597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:15.480568 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e9c144/distinct-2-w6-e9c144-kserve-77fbbdb66b-7dhxz"] Nov 02 20:18:16.183313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:16.183275 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" path="/var/lib/kubelet/pods/5d0587f2-6194-414c-a41a-ac1d9669dc0d/volumes" Nov 02 20:18:16.183789 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:16.183739 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" path="/var/lib/kubelet/pods/6cb81dd2-012d-4a42-9090-5e204dd5f18a/volumes" Nov 02 20:18:16.184237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:16.184216 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" path="/var/lib/kubelet/pods/772633d4-a7f3-47f3-a51e-fc351e9e366c/volumes" Nov 02 20:18:17.007560 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:17.007519 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" podUID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:18:18.483961 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.483925 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv"] Nov 02 20:18:18.484583 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484558 4966 cpu_manager.go:414] "containerMap: removing container" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" Nov 02 20:18:18.484682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484584 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" Nov 02 20:18:18.484682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484610 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" Nov 02 20:18:18.484682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484618 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" Nov 02 20:18:18.484682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484652 4966 cpu_manager.go:414] "containerMap: removing container" podUID="24652ba6-4d94-40f4-b85a-f00adad4183e" containerName="revoke-keys" Nov 02 20:18:18.484682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484660 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24652ba6-4d94-40f4-b85a-f00adad4183e" containerName="revoke-keys" Nov 02 20:18:18.484682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484680 4966 cpu_manager.go:414] "containerMap: removing container" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484688 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484700 4966 cpu_manager.go:414] "containerMap: removing container" podUID="45ce86f7-8400-45ce-b4cb-346fa98358f5" containerName="revoke-keys" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484709 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ce86f7-8400-45ce-b4cb-346fa98358f5" containerName="revoke-keys" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484822 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="24652ba6-4d94-40f4-b85a-f00adad4183e" containerName="revoke-keys" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484837 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="772633d4-a7f3-47f3-a51e-fc351e9e366c" containerName="main" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484849 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ce86f7-8400-45ce-b4cb-346fa98358f5" containerName="revoke-keys" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484863 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb81dd2-012d-4a42-9090-5e204dd5f18a" containerName="main" Nov 02 20:18:18.484994 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.484875 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0587f2-6194-414c-a41a-ac1d9669dc0d" containerName="main" Nov 02 20:18:18.489648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.489623 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.538776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.538745 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:18:18.584855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.584808 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:18:18.584855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.584831 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:18:18.584855 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.584857 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:18:18.588157 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.588111 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv"] Nov 02 20:18:18.606063 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.606023 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9wl\" (UniqueName: \"kubernetes.io/projected/666154df-a260-4e90-bf60-d9ad94929b90-kube-api-access-rf9wl\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.606285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.606153 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.606285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.606231 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.707811 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.707769 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.708028 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.707839 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9wl\" (UniqueName: \"kubernetes.io/projected/666154df-a260-4e90-bf60-d9ad94929b90-kube-api-access-rf9wl\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.708028 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.707890 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.708666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.708638 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.710363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.710333 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.767204 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.767116 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9wl\" (UniqueName: \"kubernetes.io/projected/666154df-a260-4e90-bf60-d9ad94929b90-kube-api-access-rf9wl\") pod \"maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:18.802267 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:18.802226 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:19.017857 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.017758 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" podUID="3ace283e-959e-4523-b74b-680ad65469bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:18:19.176530 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.176490 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv"] Nov 02 20:18:19.398088 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.398044 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt"] Nov 02 20:18:19.403469 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.403449 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.455379 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.455350 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt"] Nov 02 20:18:19.515517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.515485 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.515876 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.515604 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrxl2\" (UniqueName: \"kubernetes.io/projected/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-kube-api-access-wrxl2\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.515876 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.515641 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.617013 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.616922 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.617195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.617037 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrxl2\" (UniqueName: \"kubernetes.io/projected/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-kube-api-access-wrxl2\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.617195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.617067 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.618014 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.617987 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.619618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.619592 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.671648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.671619 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrxl2\" (UniqueName: \"kubernetes.io/projected/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-kube-api-access-wrxl2\") pod \"maas-api-revoke-sub-7478163d-w4sxt\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:19.717564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:19.717533 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:20.088052 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:20.088019 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt"] Nov 02 20:18:20.105281 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:18:20.105233 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3a7500_a873_4ca7_a6dc_32dd12b7d049.slice/crio-4c079ee8307959b9b88b2c132f361921a5498b637bd498b335df4d3b70b6724f WatchSource:0}: Error finding container 4c079ee8307959b9b88b2c132f361921a5498b637bd498b335df4d3b70b6724f: Status 404 returned error can't find the container with id 4c079ee8307959b9b88b2c132f361921a5498b637bd498b335df4d3b70b6724f Nov 02 20:18:20.155564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:20.155530 4966 generic.go:356] "Generic (PLEG): container finished" podID="666154df-a260-4e90-bf60-d9ad94929b90" containerID="ca9e271df50b8df737a92dc5e02abe0add64d454f4afee02de6e90eaa9b7d3f0" exitCode=0 Nov 02 20:18:20.155722 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:20.155615 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" event={"ID":"666154df-a260-4e90-bf60-d9ad94929b90","Type":"ContainerDied","Data":"ca9e271df50b8df737a92dc5e02abe0add64d454f4afee02de6e90eaa9b7d3f0"} Nov 02 20:18:20.155722 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:20.155671 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" event={"ID":"666154df-a260-4e90-bf60-d9ad94929b90","Type":"ContainerStarted","Data":"c40f6c8234a40054bbaf6a4fe3896e6650f98ac7f5d72a19193397e505580d46"} Nov 02 20:18:20.157708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:20.157668 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" event={"ID":"4c3a7500-a873-4ca7-a6dc-32dd12b7d049","Type":"ContainerStarted","Data":"4c079ee8307959b9b88b2c132f361921a5498b637bd498b335df4d3b70b6724f"} Nov 02 20:18:21.167657 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.167618 4966 generic.go:356] "Generic (PLEG): container finished" podID="4c3a7500-a873-4ca7-a6dc-32dd12b7d049" containerID="1bd1b61bc9085c7603d243a37bb8a3c458667fb475a54d9d9e7853d2e49f0e8d" exitCode=0 Nov 02 20:18:21.168123 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.167704 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" event={"ID":"4c3a7500-a873-4ca7-a6dc-32dd12b7d049","Type":"ContainerDied","Data":"1bd1b61bc9085c7603d243a37bb8a3c458667fb475a54d9d9e7853d2e49f0e8d"} Nov 02 20:18:21.313601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.313578 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:21.437342 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.437286 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-openshift-ca\") pod \"666154df-a260-4e90-bf60-d9ad94929b90\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " Nov 02 20:18:21.437562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.437385 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-xks-ca\") pod \"666154df-a260-4e90-bf60-d9ad94929b90\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " Nov 02 20:18:21.437562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.437442 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666154df-a260-4e90-bf60-d9ad94929b90-kube-api-access-rf9wl\" (UniqueName: \"kubernetes.io/projected/666154df-a260-4e90-bf60-d9ad94929b90-kube-api-access-rf9wl\") pod \"666154df-a260-4e90-bf60-d9ad94929b90\" (UID: \"666154df-a260-4e90-bf60-d9ad94929b90\") " Nov 02 20:18:21.438058 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.438030 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-openshift-ca" pod "666154df-a260-4e90-bf60-d9ad94929b90" (UID: "666154df-a260-4e90-bf60-d9ad94929b90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:21.440279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.440248 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-xks-ca" pod "666154df-a260-4e90-bf60-d9ad94929b90" (UID: "666154df-a260-4e90-bf60-d9ad94929b90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:21.441040 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.441013 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666154df-a260-4e90-bf60-d9ad94929b90-kube-api-access-rf9wl" pod "666154df-a260-4e90-bf60-d9ad94929b90" (UID: "666154df-a260-4e90-bf60-d9ad94929b90"). InnerVolumeSpecName "kube-api-access-rf9wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:21.539073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.539033 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:21.539073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.539067 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666154df-a260-4e90-bf60-d9ad94929b90-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:21.539073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:21.539079 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9wl\" (UniqueName: \"kubernetes.io/projected/666154df-a260-4e90-bf60-d9ad94929b90-kube-api-access-rf9wl\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.176646 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.176612 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" Nov 02 20:18:22.188279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.188246 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv" event={"ID":"666154df-a260-4e90-bf60-d9ad94929b90","Type":"ContainerDied","Data":"c40f6c8234a40054bbaf6a4fe3896e6650f98ac7f5d72a19193397e505580d46"} Nov 02 20:18:22.188279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.188282 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c40f6c8234a40054bbaf6a4fe3896e6650f98ac7f5d72a19193397e505580d46" Nov 02 20:18:22.333980 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.333953 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:22.448483 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.448378 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-xks-ca\") pod \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " Nov 02 20:18:22.448483 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.448470 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-openshift-ca\") pod \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " Nov 02 20:18:22.448727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.448549 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-kube-api-access-wrxl2\" (UniqueName: \"kubernetes.io/projected/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-kube-api-access-wrxl2\") pod \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\" (UID: \"4c3a7500-a873-4ca7-a6dc-32dd12b7d049\") " Nov 02 20:18:22.448906 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.448876 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-openshift-ca" pod "4c3a7500-a873-4ca7-a6dc-32dd12b7d049" (UID: "4c3a7500-a873-4ca7-a6dc-32dd12b7d049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:22.448996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.448980 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.450595 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.450567 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-xks-ca" pod "4c3a7500-a873-4ca7-a6dc-32dd12b7d049" (UID: "4c3a7500-a873-4ca7-a6dc-32dd12b7d049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:22.450712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.450646 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-kube-api-access-wrxl2" pod "4c3a7500-a873-4ca7-a6dc-32dd12b7d049" (UID: "4c3a7500-a873-4ca7-a6dc-32dd12b7d049"). InnerVolumeSpecName "kube-api-access-wrxl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:22.550115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.550077 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:22.550115 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:22.550106 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrxl2\" (UniqueName: \"kubernetes.io/projected/4c3a7500-a873-4ca7-a6dc-32dd12b7d049-kube-api-access-wrxl2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:23.007132 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:23.007096 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6"] Nov 02 20:18:23.024914 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:23.024875 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5"] Nov 02 20:18:23.186650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:23.186611 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" Nov 02 20:18:23.186650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:23.186625 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt" event={"ID":"4c3a7500-a873-4ca7-a6dc-32dd12b7d049","Type":"ContainerDied","Data":"4c079ee8307959b9b88b2c132f361921a5498b637bd498b335df4d3b70b6724f"} Nov 02 20:18:23.187208 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:23.186676 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c079ee8307959b9b88b2c132f361921a5498b637bd498b335df4d3b70b6724f" Nov 02 20:18:23.225070 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:23.225020 4966 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3a7500_a873_4ca7_a6dc_32dd12b7d049.slice\": RecentStats: unable to find data in memory cache]" Nov 02 20:18:27.007560 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:27.007460 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" podUID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:18:29.018474 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:29.018428 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" podUID="3ace283e-959e-4523-b74b-680ad65469bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:18:30.043197 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:30.043158 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9"] Nov 02 20:18:32.042549 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.042516 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj"] Nov 02 20:18:32.087979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.087945 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9d465-vwnsj"] Nov 02 20:18:32.183006 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.182965 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f85457-8e3c-4845-962b-da05ca3822f5" path="/var/lib/kubelet/pods/83f85457-8e3c-4845-962b-da05ca3822f5/volumes" Nov 02 20:18:32.701967 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.701941 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 20:18:32.818024 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.818000 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 20:18:32.850813 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.850787 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ace283e-959e-4523-b74b-680ad65469bf-kube-api-access-l44bh\" (UniqueName: \"kubernetes.io/projected/3ace283e-959e-4523-b74b-680ad65469bf-kube-api-access-l44bh\") pod \"3ace283e-959e-4523-b74b-680ad65469bf\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " Nov 02 20:18:32.850950 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.850821 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ace283e-959e-4523-b74b-680ad65469bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ace283e-959e-4523-b74b-680ad65469bf-plugins-config-volume\") pod \"3ace283e-959e-4523-b74b-680ad65469bf\" (UID: \"3ace283e-959e-4523-b74b-680ad65469bf\") " Nov 02 20:18:32.851217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.851191 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ace283e-959e-4523-b74b-680ad65469bf-plugins-config-volume" pod "3ace283e-959e-4523-b74b-680ad65469bf" (UID: "3ace283e-959e-4523-b74b-680ad65469bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:32.852880 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.852860 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ace283e-959e-4523-b74b-680ad65469bf-kube-api-access-l44bh" pod "3ace283e-959e-4523-b74b-680ad65469bf" (UID: "3ace283e-959e-4523-b74b-680ad65469bf"). InnerVolumeSpecName "kube-api-access-l44bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:32.951919 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.951882 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-plugins-config-volume\") pod \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " Nov 02 20:18:32.952103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.951935 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-kube-api-access-6qzds\" (UniqueName: \"kubernetes.io/projected/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-kube-api-access-6qzds\") pod \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\" (UID: \"25a17621-7bdd-49aa-a4bd-5a2d4f98903a\") " Nov 02 20:18:32.952168 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.952132 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44bh\" (UniqueName: \"kubernetes.io/projected/3ace283e-959e-4523-b74b-680ad65469bf-kube-api-access-l44bh\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.952168 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.952142 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ace283e-959e-4523-b74b-680ad65469bf-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.952433 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.952406 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-plugins-config-volume" pod "25a17621-7bdd-49aa-a4bd-5a2d4f98903a" (UID: "25a17621-7bdd-49aa-a4bd-5a2d4f98903a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:32.954225 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:32.954196 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-kube-api-access-6qzds" pod "25a17621-7bdd-49aa-a4bd-5a2d4f98903a" (UID: "25a17621-7bdd-49aa-a4bd-5a2d4f98903a"). InnerVolumeSpecName "kube-api-access-6qzds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:33.052761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.052668 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.052761 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.052705 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzds\" (UniqueName: \"kubernetes.io/projected/25a17621-7bdd-49aa-a4bd-5a2d4f98903a-kube-api-access-6qzds\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.255149 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.255116 4966 generic.go:356] "Generic (PLEG): container finished" podID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" containerID="410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422" exitCode=0 Nov 02 20:18:33.255349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.255217 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" Nov 02 20:18:33.255349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.255258 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" event={"ID":"25a17621-7bdd-49aa-a4bd-5a2d4f98903a","Type":"ContainerDied","Data":"410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422"} Nov 02 20:18:33.255349 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.255327 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv" event={"ID":"25a17621-7bdd-49aa-a4bd-5a2d4f98903a","Type":"ContainerDied","Data":"865d9e5d4df6f6f5ec2e325cd4c80ce9b1d387c260432c8022e68536682fa2d1"} Nov 02 20:18:33.255518 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.255357 4966 scope.go:122] "RemoveContainer" containerID="410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422" Nov 02 20:18:33.256931 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.256899 4966 generic.go:356] "Generic (PLEG): container finished" podID="3ace283e-959e-4523-b74b-680ad65469bf" containerID="b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736" exitCode=0 Nov 02 20:18:33.257070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.256945 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" event={"ID":"3ace283e-959e-4523-b74b-680ad65469bf","Type":"ContainerDied","Data":"b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736"} Nov 02 20:18:33.257070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.256950 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" Nov 02 20:18:33.257070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.256984 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49" event={"ID":"3ace283e-959e-4523-b74b-680ad65469bf","Type":"ContainerDied","Data":"1d2363ad29c55cf4ea57ee4ed7517b36e8b6e30358f031c3ed858d5eeffe807d"} Nov 02 20:18:33.267012 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.266991 4966 scope.go:122] "RemoveContainer" containerID="410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422" Nov 02 20:18:33.267328 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:33.267276 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422\": container with ID starting with 410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422 not found: ID does not exist" containerID="410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422" Nov 02 20:18:33.267427 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.267328 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422"} err="failed to get container status \"410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422\": rpc error: code = NotFound desc = could not find container \"410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422\": container with ID starting with 410d8528c93061733a26a04bf547c18f91017fecac1997fdea8ecff368da1422 not found: ID does not exist" Nov 02 20:18:33.267427 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.267350 4966 scope.go:122] "RemoveContainer" containerID="b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736" Nov 02 20:18:33.278289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.278267 4966 scope.go:122] "RemoveContainer" containerID="b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736" Nov 02 20:18:33.278595 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:33.278572 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736\": container with ID starting with b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736 not found: ID does not exist" containerID="b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736" Nov 02 20:18:33.278660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.278604 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736"} err="failed to get container status \"b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736\": rpc error: code = NotFound desc = could not find container \"b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736\": container with ID starting with b944b6d7b9edcfb6d043b96fc3ff7516722270aa63969bf9f66af727b7a88736 not found: ID does not exist" Nov 02 20:18:33.286829 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:33.286808 4966 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ace283e_959e_4523_b74b_680ad65469bf.slice/crio-1d2363ad29c55cf4ea57ee4ed7517b36e8b6e30358f031c3ed858d5eeffe807d\": RecentStats: unable to find data in memory cache]" Nov 02 20:18:33.408841 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.408757 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49"] Nov 02 20:18:33.454673 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.454635 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e9c144-cdb96b99-dvn49"] Nov 02 20:18:33.555252 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.555218 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv"] Nov 02 20:18:33.559004 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:33.558971 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e9c144-5498b87467-lpktv"] Nov 02 20:18:34.182506 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:34.182472 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" path="/var/lib/kubelet/pods/25a17621-7bdd-49aa-a4bd-5a2d4f98903a/volumes" Nov 02 20:18:34.182920 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:34.182898 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ace283e-959e-4523-b74b-680ad65469bf" path="/var/lib/kubelet/pods/3ace283e-959e-4523-b74b-680ad65469bf/volumes" Nov 02 20:18:35.044561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.044510 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" podUID="93c29350-0a6c-48cd-abfd-7d55a3903c34" containerName="payload-processing" containerID="cri-o://55c348d1e3469e1abd106b4ba06bce663dfaf9ad854712ea7c33567b0d9c09c0" gracePeriod=25 Nov 02 20:18:35.273316 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.273274 4966 generic.go:356] "Generic (PLEG): container finished" podID="93c29350-0a6c-48cd-abfd-7d55a3903c34" containerID="55c348d1e3469e1abd106b4ba06bce663dfaf9ad854712ea7c33567b0d9c09c0" exitCode=0 Nov 02 20:18:35.273684 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.273328 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" event={"ID":"93c29350-0a6c-48cd-abfd-7d55a3903c34","Type":"ContainerDied","Data":"55c348d1e3469e1abd106b4ba06bce663dfaf9ad854712ea7c33567b0d9c09c0"} Nov 02 20:18:35.303265 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.303205 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 20:18:35.375620 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.375582 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c29350-0a6c-48cd-abfd-7d55a3903c34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c29350-0a6c-48cd-abfd-7d55a3903c34-plugins-config-volume\") pod \"93c29350-0a6c-48cd-abfd-7d55a3903c34\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " Nov 02 20:18:35.375812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.375713 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c29350-0a6c-48cd-abfd-7d55a3903c34-kube-api-access-44w8z\" (UniqueName: \"kubernetes.io/projected/93c29350-0a6c-48cd-abfd-7d55a3903c34-kube-api-access-44w8z\") pod \"93c29350-0a6c-48cd-abfd-7d55a3903c34\" (UID: \"93c29350-0a6c-48cd-abfd-7d55a3903c34\") " Nov 02 20:18:35.376069 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.376046 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c29350-0a6c-48cd-abfd-7d55a3903c34-plugins-config-volume" pod "93c29350-0a6c-48cd-abfd-7d55a3903c34" (UID: "93c29350-0a6c-48cd-abfd-7d55a3903c34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:35.377905 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.377877 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c29350-0a6c-48cd-abfd-7d55a3903c34-kube-api-access-44w8z" pod "93c29350-0a6c-48cd-abfd-7d55a3903c34" (UID: "93c29350-0a6c-48cd-abfd-7d55a3903c34"). InnerVolumeSpecName "kube-api-access-44w8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:35.477029 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.476992 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44w8z\" (UniqueName: \"kubernetes.io/projected/93c29350-0a6c-48cd-abfd-7d55a3903c34-kube-api-access-44w8z\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:35.477374 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:35.477351 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c29350-0a6c-48cd-abfd-7d55a3903c34-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:36.284643 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:36.284599 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" event={"ID":"93c29350-0a6c-48cd-abfd-7d55a3903c34","Type":"ContainerDied","Data":"7b3fcc06132ebf91d560b3a20736e6c7f144ad1d0b890a821a748d29d59965da"} Nov 02 20:18:36.285166 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:36.284684 4966 scope.go:122] "RemoveContainer" containerID="55c348d1e3469e1abd106b4ba06bce663dfaf9ad854712ea7c33567b0d9c09c0" Nov 02 20:18:36.285166 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:36.284835 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9" Nov 02 20:18:36.437946 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:36.437913 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9"] Nov 02 20:18:36.442821 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:36.442753 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-26cfbe-75d8dbcd5c-nt2t9"] Nov 02 20:18:37.478056 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478021 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd"] Nov 02 20:18:37.478495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478480 4966 cpu_manager.go:414] "containerMap: removing container" podUID="4c3a7500-a873-4ca7-a6dc-32dd12b7d049" containerName="revoke-keys" Nov 02 20:18:37.478545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478497 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3a7500-a873-4ca7-a6dc-32dd12b7d049" containerName="revoke-keys" Nov 02 20:18:37.478545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478509 4966 cpu_manager.go:414] "containerMap: removing container" podUID="666154df-a260-4e90-bf60-d9ad94929b90" containerName="revoke-keys" Nov 02 20:18:37.478545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478514 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666154df-a260-4e90-bf60-d9ad94929b90" containerName="revoke-keys" Nov 02 20:18:37.478545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478525 4966 cpu_manager.go:414] "containerMap: removing container" podUID="93c29350-0a6c-48cd-abfd-7d55a3903c34" containerName="payload-processing" Nov 02 20:18:37.478545 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478534 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c29350-0a6c-48cd-abfd-7d55a3903c34" containerName="payload-processing" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478552 4966 cpu_manager.go:414] "containerMap: removing container" podUID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" containerName="payload-pre-processing" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478558 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" containerName="payload-pre-processing" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478566 4966 cpu_manager.go:414] "containerMap: removing container" podUID="3ace283e-959e-4523-b74b-680ad65469bf" containerName="payload-processing" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478572 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ace283e-959e-4523-b74b-680ad65469bf" containerName="payload-processing" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478637 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3a7500-a873-4ca7-a6dc-32dd12b7d049" containerName="revoke-keys" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478647 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="666154df-a260-4e90-bf60-d9ad94929b90" containerName="revoke-keys" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478654 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c29350-0a6c-48cd-abfd-7d55a3903c34" containerName="payload-processing" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478662 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a17621-7bdd-49aa-a4bd-5a2d4f98903a" containerName="payload-pre-processing" Nov 02 20:18:37.478796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.478668 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ace283e-959e-4523-b74b-680ad65469bf" containerName="payload-processing" Nov 02 20:18:37.483499 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.483472 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.534984 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.534953 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd"] Nov 02 20:18:37.577458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.577430 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:18:37.577458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.577434 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:18:37.577666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.577435 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:18:37.577666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.577499 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:18:37.596950 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.596922 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.597120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.597068 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.597192 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.597147 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtkr\" (UniqueName: \"kubernetes.io/projected/0cc31d5c-545a-4047-9e1d-e2c562d7808b-kube-api-access-ljtkr\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.698366 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.698330 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtkr\" (UniqueName: \"kubernetes.io/projected/0cc31d5c-545a-4047-9e1d-e2c562d7808b-kube-api-access-ljtkr\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.698572 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.698392 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.698572 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.698499 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.699277 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.699239 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.701031 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.701002 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.751968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.751893 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtkr\" (UniqueName: \"kubernetes.io/projected/0cc31d5c-545a-4047-9e1d-e2c562d7808b-kube-api-access-ljtkr\") pod \"maas-api-revoke-sub-be5d1f7b-mqsrd\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:37.798661 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:37.798625 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:38.027836 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.027732 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" containerID="cri-o://526d625524b07af010225b37a324e743092016340a94186d6ef3dadd44842a20" gracePeriod=45 Nov 02 20:18:38.044519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.044478 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" containerID="cri-o://6c13f9e0f78b3a3adfdf428b2d564b20d0e4bdf52a58579bce429fb31b5c9942" gracePeriod=45 Nov 02 20:18:38.174674 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.174635 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd"] Nov 02 20:18:38.186070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.186045 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c29350-0a6c-48cd-abfd-7d55a3903c34" path="/var/lib/kubelet/pods/93c29350-0a6c-48cd-abfd-7d55a3903c34/volumes" Nov 02 20:18:38.306970 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.306939 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 20:18:38.317553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.317488 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" event={"ID":"0cc31d5c-545a-4047-9e1d-e2c562d7808b","Type":"ContainerStarted","Data":"3576d830e7bce7b258366aa2915d0b51d3e325d9ed421e19d12d579df2717579"} Nov 02 20:18:38.319492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.319464 4966 generic.go:356] "Generic (PLEG): container finished" podID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerID="6c13f9e0f78b3a3adfdf428b2d564b20d0e4bdf52a58579bce429fb31b5c9942" exitCode=0 Nov 02 20:18:38.319633 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.319598 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" event={"ID":"58bc87bd-519d-4b6e-b6bf-199749452d94","Type":"ContainerDied","Data":"6c13f9e0f78b3a3adfdf428b2d564b20d0e4bdf52a58579bce429fb31b5c9942"} Nov 02 20:18:38.321570 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.321408 4966 generic.go:356] "Generic (PLEG): container finished" podID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerID="526d625524b07af010225b37a324e743092016340a94186d6ef3dadd44842a20" exitCode=0 Nov 02 20:18:38.321570 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.321459 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" event={"ID":"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda","Type":"ContainerDied","Data":"526d625524b07af010225b37a324e743092016340a94186d6ef3dadd44842a20"} Nov 02 20:18:38.321570 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.321515 4966 scope.go:122] "RemoveContainer" containerID="526d625524b07af010225b37a324e743092016340a94186d6ef3dadd44842a20" Nov 02 20:18:38.321570 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.321471 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6" Nov 02 20:18:38.373314 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.373273 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 20:18:38.406122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406074 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-home\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-home\") pod \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " Nov 02 20:18:38.406326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406135 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tmp-dir\") pod \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " Nov 02 20:18:38.406326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406201 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tls-certs\" (UniqueName: \"kubernetes.io/secret/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tls-certs\") pod \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " Nov 02 20:18:38.406326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406251 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-kube-api-access-grb4q\" (UniqueName: \"kubernetes.io/projected/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-kube-api-access-grb4q\") pod \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " Nov 02 20:18:38.406527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406348 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-dshm\") pod \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " Nov 02 20:18:38.406527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406380 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-model-cache\") pod \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\" (UID: \"d80b1bff-24ef-4e5c-bd1e-4b57a1654eda\") " Nov 02 20:18:38.406527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406385 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-home" pod "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" (UID: "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.406527 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406467 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tmp-dir" pod "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" (UID: "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.406757 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406661 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-model-cache" pod "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" (UID: "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.406812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406799 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.406864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406817 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.406864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.406834 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.408726 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.408703 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-dshm" pod "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" (UID: "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.408821 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.408735 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tls-certs" pod "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" (UID: "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:38.408821 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.408756 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-kube-api-access-grb4q" pod "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" (UID: "d80b1bff-24ef-4e5c-bd1e-4b57a1654eda"). InnerVolumeSpecName "kube-api-access-grb4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:38.507188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507157 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-model-cache\") pod \"58bc87bd-519d-4b6e-b6bf-199749452d94\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " Nov 02 20:18:38.507188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507194 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-dshm\") pod \"58bc87bd-519d-4b6e-b6bf-199749452d94\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507214 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-tmp-dir\") pod \"58bc87bd-519d-4b6e-b6bf-199749452d94\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507245 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bc87bd-519d-4b6e-b6bf-199749452d94-kube-api-access-6tpmf\" (UniqueName: \"kubernetes.io/projected/58bc87bd-519d-4b6e-b6bf-199749452d94-kube-api-access-6tpmf\") pod \"58bc87bd-519d-4b6e-b6bf-199749452d94\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507271 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-home\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-home\") pod \"58bc87bd-519d-4b6e-b6bf-199749452d94\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507361 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58bc87bd-519d-4b6e-b6bf-199749452d94-tls-certs\" (UniqueName: \"kubernetes.io/secret/58bc87bd-519d-4b6e-b6bf-199749452d94-tls-certs\") pod \"58bc87bd-519d-4b6e-b6bf-199749452d94\" (UID: \"58bc87bd-519d-4b6e-b6bf-199749452d94\") " Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507494 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-model-cache" pod "58bc87bd-519d-4b6e-b6bf-199749452d94" (UID: "58bc87bd-519d-4b6e-b6bf-199749452d94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507588 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-tmp-dir" pod "58bc87bd-519d-4b6e-b6bf-199749452d94" (UID: "58bc87bd-519d-4b6e-b6bf-199749452d94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507682 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507701 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507716 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507730 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grb4q\" (UniqueName: \"kubernetes.io/projected/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-kube-api-access-grb4q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507744 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.507765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.507760 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-home" pod "58bc87bd-519d-4b6e-b6bf-199749452d94" (UID: "58bc87bd-519d-4b6e-b6bf-199749452d94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.511642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.511258 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-dshm" pod "58bc87bd-519d-4b6e-b6bf-199749452d94" (UID: "58bc87bd-519d-4b6e-b6bf-199749452d94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:38.511642 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.511377 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bc87bd-519d-4b6e-b6bf-199749452d94-tls-certs" pod "58bc87bd-519d-4b6e-b6bf-199749452d94" (UID: "58bc87bd-519d-4b6e-b6bf-199749452d94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:38.511851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.511633 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bc87bd-519d-4b6e-b6bf-199749452d94-kube-api-access-6tpmf" pod "58bc87bd-519d-4b6e-b6bf-199749452d94" (UID: "58bc87bd-519d-4b6e-b6bf-199749452d94"). InnerVolumeSpecName "kube-api-access-6tpmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:38.609446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.609332 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tpmf\" (UniqueName: \"kubernetes.io/projected/58bc87bd-519d-4b6e-b6bf-199749452d94-kube-api-access-6tpmf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.609446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.609383 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.609446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.609398 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58bc87bd-519d-4b6e-b6bf-199749452d94-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.609446 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.609411 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58bc87bd-519d-4b6e-b6bf-199749452d94-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:38.736223 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.736187 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6"] Nov 02 20:18:38.742662 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:38.742629 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/distinct-2-w1-26cfbe-kserve-9c8f89ff9-p4zw6"] Nov 02 20:18:39.331102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.331062 4966 generic.go:356] "Generic (PLEG): container finished" podID="0cc31d5c-545a-4047-9e1d-e2c562d7808b" containerID="7033f18f7dd5184bd6d88f4064b138a3584ea592afa4acca10d1aac919928689" exitCode=0 Nov 02 20:18:39.331311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.331155 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" event={"ID":"0cc31d5c-545a-4047-9e1d-e2c562d7808b","Type":"ContainerDied","Data":"7033f18f7dd5184bd6d88f4064b138a3584ea592afa4acca10d1aac919928689"} Nov 02 20:18:39.333029 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.333006 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" event={"ID":"58bc87bd-519d-4b6e-b6bf-199749452d94","Type":"ContainerDied","Data":"440e121123fd2b37ade3f18a9130c84f76be6b5f39a75f47bc2aeca5ddddb072"} Nov 02 20:18:39.333151 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.333045 4966 scope.go:122] "RemoveContainer" containerID="6c13f9e0f78b3a3adfdf428b2d564b20d0e4bdf52a58579bce429fb31b5c9942" Nov 02 20:18:39.333151 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.333050 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5" Nov 02 20:18:39.540500 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.540464 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5"] Nov 02 20:18:39.544254 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.544224 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26cfbe/premium-w1-26cfbe-kserve-64468758f-z2bj5"] Nov 02 20:18:39.576262 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.576232 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p"] Nov 02 20:18:39.586249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:39.586224 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e9c144-sj78p"] Nov 02 20:18:40.182650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.182614 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ce86f7-8400-45ce-b4cb-346fa98358f5" path="/var/lib/kubelet/pods/45ce86f7-8400-45ce-b4cb-346fa98358f5/volumes" Nov 02 20:18:40.183145 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.183127 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" path="/var/lib/kubelet/pods/58bc87bd-519d-4b6e-b6bf-199749452d94/volumes" Nov 02 20:18:40.183574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.183560 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" path="/var/lib/kubelet/pods/d80b1bff-24ef-4e5c-bd1e-4b57a1654eda/volumes" Nov 02 20:18:40.510768 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.510738 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:40.629167 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.629132 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc31d5c-545a-4047-9e1d-e2c562d7808b-kube-api-access-ljtkr\" (UniqueName: \"kubernetes.io/projected/0cc31d5c-545a-4047-9e1d-e2c562d7808b-kube-api-access-ljtkr\") pod \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " Nov 02 20:18:40.629648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.629248 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-openshift-ca\") pod \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " Nov 02 20:18:40.629648 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.629275 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-xks-ca\") pod \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\" (UID: \"0cc31d5c-545a-4047-9e1d-e2c562d7808b\") " Nov 02 20:18:40.629751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.629686 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-openshift-ca" pod "0cc31d5c-545a-4047-9e1d-e2c562d7808b" (UID: "0cc31d5c-545a-4047-9e1d-e2c562d7808b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:40.631339 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.631284 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-xks-ca" pod "0cc31d5c-545a-4047-9e1d-e2c562d7808b" (UID: "0cc31d5c-545a-4047-9e1d-e2c562d7808b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:40.631470 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.631417 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc31d5c-545a-4047-9e1d-e2c562d7808b-kube-api-access-ljtkr" pod "0cc31d5c-545a-4047-9e1d-e2c562d7808b" (UID: "0cc31d5c-545a-4047-9e1d-e2c562d7808b"). InnerVolumeSpecName "kube-api-access-ljtkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:40.730622 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.730584 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtkr\" (UniqueName: \"kubernetes.io/projected/0cc31d5c-545a-4047-9e1d-e2c562d7808b-kube-api-access-ljtkr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:40.730622 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.730616 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:40.730622 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:40.730627 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc31d5c-545a-4047-9e1d-e2c562d7808b-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:41.351949 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:41.351914 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" Nov 02 20:18:41.351949 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:41.351938 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd" event={"ID":"0cc31d5c-545a-4047-9e1d-e2c562d7808b","Type":"ContainerDied","Data":"3576d830e7bce7b258366aa2915d0b51d3e325d9ed421e19d12d579df2717579"} Nov 02 20:18:41.352231 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:41.351992 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3576d830e7bce7b258366aa2915d0b51d3e325d9ed421e19d12d579df2717579" Nov 02 20:18:44.752356 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:18:44.752319 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:18:45.044000 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:45.043915 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft"] Nov 02 20:18:45.089920 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:45.089867 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b751c2-28kft"] Nov 02 20:18:46.183137 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:46.183099 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621b4a1e-76f1-4766-9429-fafd758c4d0b" path="/var/lib/kubelet/pods/621b4a1e-76f1-4766-9429-fafd758c4d0b/volumes" Nov 02 20:18:55.521035 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:55.520995 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv"] Nov 02 20:18:55.566598 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:55.566565 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-26cfbe-pdlfv"] Nov 02 20:18:56.182456 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:18:56.182419 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666154df-a260-4e90-bf60-d9ad94929b90" path="/var/lib/kubelet/pods/666154df-a260-4e90-bf60-d9ad94929b90/volumes" Nov 02 20:19:02.069627 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:02.069592 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb"] Nov 02 20:19:02.104748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:02.104715 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14a67b71-mf6lb"] Nov 02 20:19:02.183030 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:02.182998 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5056aa2-c2e7-4fd4-8463-51d901ae282f" path="/var/lib/kubelet/pods/f5056aa2-c2e7-4fd4-8463-51d901ae282f/volumes" Nov 02 20:19:07.782041 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:07.782003 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-t47p9"] Nov 02 20:19:07.782534 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:07.782221 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" podUID="999c00dd-b142-421a-a171-a7185d979eeb" containerName="manager" containerID="cri-o://7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6" gracePeriod=30 Nov 02 20:19:08.033094 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.033025 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 20:19:08.073435 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.073404 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999c00dd-b142-421a-a171-a7185d979eeb-kube-api-access-c4h7c\" (UniqueName: \"kubernetes.io/projected/999c00dd-b142-421a-a171-a7185d979eeb-kube-api-access-c4h7c\") pod \"999c00dd-b142-421a-a171-a7185d979eeb\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " Nov 02 20:19:08.073435 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.073439 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-metrics-certs\") pod \"999c00dd-b142-421a-a171-a7185d979eeb\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " Nov 02 20:19:08.073647 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.073508 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-webhook-cert\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-webhook-cert\") pod \"999c00dd-b142-421a-a171-a7185d979eeb\" (UID: \"999c00dd-b142-421a-a171-a7185d979eeb\") " Nov 02 20:19:08.075675 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.075636 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-metrics-certs" pod "999c00dd-b142-421a-a171-a7185d979eeb" (UID: "999c00dd-b142-421a-a171-a7185d979eeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:19:08.075829 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.075682 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-webhook-cert" pod "999c00dd-b142-421a-a171-a7185d979eeb" (UID: "999c00dd-b142-421a-a171-a7185d979eeb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:19:08.075829 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.075682 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999c00dd-b142-421a-a171-a7185d979eeb-kube-api-access-c4h7c" pod "999c00dd-b142-421a-a171-a7185d979eeb" (UID: "999c00dd-b142-421a-a171-a7185d979eeb"). InnerVolumeSpecName "kube-api-access-c4h7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:08.174249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.174203 4966 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-webhook-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:08.174249 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.174247 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4h7c\" (UniqueName: \"kubernetes.io/projected/999c00dd-b142-421a-a171-a7185d979eeb-kube-api-access-c4h7c\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:08.174471 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.174264 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999c00dd-b142-421a-a171-a7185d979eeb-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:08.536791 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.536759 4966 generic.go:356] "Generic (PLEG): container finished" podID="999c00dd-b142-421a-a171-a7185d979eeb" containerID="7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6" exitCode=0 Nov 02 20:19:08.536972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.536803 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" Nov 02 20:19:08.536972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.536843 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" event={"ID":"999c00dd-b142-421a-a171-a7185d979eeb","Type":"ContainerDied","Data":"7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6"} Nov 02 20:19:08.536972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.536902 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-t47p9" event={"ID":"999c00dd-b142-421a-a171-a7185d979eeb","Type":"ContainerDied","Data":"181bc22ceaed24734ab3c96264c75015c3fd1d459d5907b576912caa20b2140e"} Nov 02 20:19:08.536972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.536930 4966 scope.go:122] "RemoveContainer" containerID="7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6" Nov 02 20:19:08.548043 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.548004 4966 scope.go:122] "RemoveContainer" containerID="7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6" Nov 02 20:19:08.548281 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:19:08.548260 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6\": container with ID starting with 7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6 not found: ID does not exist" containerID="7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6" Nov 02 20:19:08.548387 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.548292 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6"} err="failed to get container status \"7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6\": rpc error: code = NotFound desc = could not find container \"7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6\": container with ID starting with 7fb69d05e91c5183ce0b3e3d38ffe7e44cfb34649e2cf84d7bd5973574abf3d6 not found: ID does not exist" Nov 02 20:19:08.689777 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.689736 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-t47p9"] Nov 02 20:19:08.735678 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:08.735647 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-t47p9"] Nov 02 20:19:09.554796 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.554761 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-tdtgx"] Nov 02 20:19:09.555354 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555336 4966 cpu_manager.go:414] "containerMap: removing container" podUID="999c00dd-b142-421a-a171-a7185d979eeb" containerName="manager" Nov 02 20:19:09.555418 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555357 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c00dd-b142-421a-a171-a7185d979eeb" containerName="manager" Nov 02 20:19:09.555418 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555376 4966 cpu_manager.go:414] "containerMap: removing container" podUID="0cc31d5c-545a-4047-9e1d-e2c562d7808b" containerName="revoke-keys" Nov 02 20:19:09.555418 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555385 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc31d5c-545a-4047-9e1d-e2c562d7808b" containerName="revoke-keys" Nov 02 20:19:09.555523 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555418 4966 cpu_manager.go:414] "containerMap: removing container" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" Nov 02 20:19:09.555523 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555425 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" Nov 02 20:19:09.555523 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555446 4966 cpu_manager.go:414] "containerMap: removing container" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" Nov 02 20:19:09.555523 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555453 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" Nov 02 20:19:09.555650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555548 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bc87bd-519d-4b6e-b6bf-199749452d94" containerName="main" Nov 02 20:19:09.555650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555561 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80b1bff-24ef-4e5c-bd1e-4b57a1654eda" containerName="main" Nov 02 20:19:09.555650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555573 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c00dd-b142-421a-a171-a7185d979eeb" containerName="manager" Nov 02 20:19:09.555650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.555587 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc31d5c-545a-4047-9e1d-e2c562d7808b" containerName="revoke-keys" Nov 02 20:19:09.561084 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.561067 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.585405 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.585369 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.585584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.585431 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.585584 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.585523 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v62j\" (UniqueName: \"kubernetes.io/projected/0beafeca-d52b-4709-aded-85eef26e4a34-kube-api-access-8v62j\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.609961 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.609927 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 20:19:09.655839 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.655794 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 20:19:09.655839 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.655793 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gbrc7\"" Nov 02 20:19:09.655839 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.655808 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 20:19:09.661002 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.660970 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-tdtgx"] Nov 02 20:19:09.686944 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.686912 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.687140 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.686959 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.687140 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.687075 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v62j\" (UniqueName: \"kubernetes.io/projected/0beafeca-d52b-4709-aded-85eef26e4a34-kube-api-access-8v62j\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.689460 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.689430 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.689604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.689515 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.739861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.739821 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v62j\" (UniqueName: \"kubernetes.io/projected/0beafeca-d52b-4709-aded-85eef26e4a34-kube-api-access-8v62j\") pod \"maas-controller-6ddc6f7dd5-tdtgx\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:09.874335 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:09.874231 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:10.184198 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:10.184109 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999c00dd-b142-421a-a171-a7185d979eeb" path="/var/lib/kubelet/pods/999c00dd-b142-421a-a171-a7185d979eeb/volumes" Nov 02 20:19:10.254233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:10.254199 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-tdtgx"] Nov 02 20:19:10.257800 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:19:10.257757 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0beafeca_d52b_4709_aded_85eef26e4a34.slice/crio-76b1d80876abca06ddbc1cab36f25c66e8f35cd1c7b024f59602cb3b11c2a854 WatchSource:0}: Error finding container 76b1d80876abca06ddbc1cab36f25c66e8f35cd1c7b024f59602cb3b11c2a854: Status 404 returned error can't find the container with id 76b1d80876abca06ddbc1cab36f25c66e8f35cd1c7b024f59602cb3b11c2a854 Nov 02 20:19:10.259324 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:10.259273 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:19:10.564623 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:10.564586 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" event={"ID":"0beafeca-d52b-4709-aded-85eef26e4a34","Type":"ContainerStarted","Data":"76b1d80876abca06ddbc1cab36f25c66e8f35cd1c7b024f59602cb3b11c2a854"} Nov 02 20:19:11.578562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:11.578466 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" event={"ID":"0beafeca-d52b-4709-aded-85eef26e4a34","Type":"ContainerStarted","Data":"99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9"} Nov 02 20:19:11.579195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:11.579157 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:11.710284 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:11.710215 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" podStartSLOduration=1.7032474560000002 podStartE2EDuration="2.71019241s" totalImagesPullingTime="1.006944954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:11.705386138 +0000 UTC m=+3057.654041976" watchObservedRunningTime="2026-11-02 20:19:11.71019241 +0000 UTC m=+3057.658848250" Nov 02 20:19:14.752778 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:19:14.752738 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:19:19.335326 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:19.335283 4966 scope.go:122] "RemoveContainer" containerID="d342671d383ae933b3cd567123a35e35b7ca98bd35bdfcd571e1591d6dd3cd92" Nov 02 20:19:22.588444 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:22.588409 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:28.042194 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:28.042153 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h"] Nov 02 20:19:28.087866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:28.087821 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70e5aed6-lvj9h"] Nov 02 20:19:28.182044 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:28.182009 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce845758-965b-496e-94a1-35908a53459d" path="/var/lib/kubelet/pods/ce845758-965b-496e-94a1-35908a53459d/volumes" Nov 02 20:19:40.043346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:40.043293 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw"] Nov 02 20:19:40.089841 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:40.089803 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc8d2e-qxwvw"] Nov 02 20:19:40.182232 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:40.182194 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e2ed6b-8546-43a0-9207-e8f97d24aa84" path="/var/lib/kubelet/pods/16e2ed6b-8546-43a0-9207-e8f97d24aa84/volumes" Nov 02 20:19:43.759812 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:43.759777 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-tdtgx"] Nov 02 20:19:43.760283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:43.759964 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" podUID="0beafeca-d52b-4709-aded-85eef26e4a34" containerName="manager" containerID="cri-o://99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9" gracePeriod=30 Nov 02 20:19:44.012243 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.012176 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:44.079630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.079597 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-webhook-cert\") pod \"0beafeca-d52b-4709-aded-85eef26e4a34\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " Nov 02 20:19:44.079856 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.079656 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0beafeca-d52b-4709-aded-85eef26e4a34-kube-api-access-8v62j\" (UniqueName: \"kubernetes.io/projected/0beafeca-d52b-4709-aded-85eef26e4a34-kube-api-access-8v62j\") pod \"0beafeca-d52b-4709-aded-85eef26e4a34\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " Nov 02 20:19:44.079856 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.079773 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-metrics-certs\") pod \"0beafeca-d52b-4709-aded-85eef26e4a34\" (UID: \"0beafeca-d52b-4709-aded-85eef26e4a34\") " Nov 02 20:19:44.081830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.081800 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-metrics-certs" pod "0beafeca-d52b-4709-aded-85eef26e4a34" (UID: "0beafeca-d52b-4709-aded-85eef26e4a34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:19:44.081951 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.081828 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-webhook-cert" pod "0beafeca-d52b-4709-aded-85eef26e4a34" (UID: "0beafeca-d52b-4709-aded-85eef26e4a34"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:19:44.081951 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.081920 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0beafeca-d52b-4709-aded-85eef26e4a34-kube-api-access-8v62j" pod "0beafeca-d52b-4709-aded-85eef26e4a34" (UID: "0beafeca-d52b-4709-aded-85eef26e4a34"). InnerVolumeSpecName "kube-api-access-8v62j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.180735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.180701 4966 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.180735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.180731 4966 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0beafeca-d52b-4709-aded-85eef26e4a34-webhook-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.180735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.180742 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v62j\" (UniqueName: \"kubernetes.io/projected/0beafeca-d52b-4709-aded-85eef26e4a34-kube-api-access-8v62j\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.751603 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:19:44.751564 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:19:44.803708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.803667 4966 generic.go:356] "Generic (PLEG): container finished" podID="0beafeca-d52b-4709-aded-85eef26e4a34" containerID="99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9" exitCode=0 Nov 02 20:19:44.804237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.803733 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" Nov 02 20:19:44.804237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.803752 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" event={"ID":"0beafeca-d52b-4709-aded-85eef26e4a34","Type":"ContainerDied","Data":"99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9"} Nov 02 20:19:44.804237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.803808 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-tdtgx" event={"ID":"0beafeca-d52b-4709-aded-85eef26e4a34","Type":"ContainerDied","Data":"76b1d80876abca06ddbc1cab36f25c66e8f35cd1c7b024f59602cb3b11c2a854"} Nov 02 20:19:44.804237 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.803831 4966 scope.go:122] "RemoveContainer" containerID="99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9" Nov 02 20:19:44.814254 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.814233 4966 scope.go:122] "RemoveContainer" containerID="99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9" Nov 02 20:19:44.814560 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:19:44.814534 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9\": container with ID starting with 99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9 not found: ID does not exist" containerID="99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9" Nov 02 20:19:44.814672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.814570 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9"} err="failed to get container status \"99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9\": rpc error: code = NotFound desc = could not find container \"99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9\": container with ID starting with 99c5a360780976141e35bcc6725f5ceeab4faf9cea09d20fbbc28c2f232e9da9 not found: ID does not exist" Nov 02 20:19:44.956357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:44.956319 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-tdtgx"] Nov 02 20:19:45.002747 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:45.002672 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-tdtgx"] Nov 02 20:19:46.182092 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.182058 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0beafeca-d52b-4709-aded-85eef26e4a34" path="/var/lib/kubelet/pods/0beafeca-d52b-4709-aded-85eef26e4a34/volumes" Nov 02 20:19:46.293268 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.293233 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-p4d2c"] Nov 02 20:19:46.293755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.293736 4966 cpu_manager.go:414] "containerMap: removing container" podUID="0beafeca-d52b-4709-aded-85eef26e4a34" containerName="manager" Nov 02 20:19:46.293819 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.293758 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0beafeca-d52b-4709-aded-85eef26e4a34" containerName="manager" Nov 02 20:19:46.293861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.293838 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0beafeca-d52b-4709-aded-85eef26e4a34" containerName="manager" Nov 02 20:19:46.298330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.298292 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.393112 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.393066 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gbrc7\"" Nov 02 20:19:46.393367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.393346 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 20:19:46.393450 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.393380 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 20:19:46.393450 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.393437 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 20:19:46.396591 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.396565 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-p4d2c"] Nov 02 20:19:46.402668 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.402640 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcedd47a-6389-43d4-9765-b0a34bfe88f6-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.402801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.402697 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcedd47a-6389-43d4-9765-b0a34bfe88f6-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.402801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.402788 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2vt\" (UniqueName: \"kubernetes.io/projected/bcedd47a-6389-43d4-9765-b0a34bfe88f6-kube-api-access-ln2vt\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.503905 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.503864 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcedd47a-6389-43d4-9765-b0a34bfe88f6-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.504111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.503914 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2vt\" (UniqueName: \"kubernetes.io/projected/bcedd47a-6389-43d4-9765-b0a34bfe88f6-kube-api-access-ln2vt\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.504111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.503986 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcedd47a-6389-43d4-9765-b0a34bfe88f6-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.506443 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.506418 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcedd47a-6389-43d4-9765-b0a34bfe88f6-webhook-cert\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.506659 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.506638 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcedd47a-6389-43d4-9765-b0a34bfe88f6-metrics-certs\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.556917 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.556887 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2vt\" (UniqueName: \"kubernetes.io/projected/bcedd47a-6389-43d4-9765-b0a34bfe88f6-kube-api-access-ln2vt\") pod \"maas-controller-6ddc6f7dd5-p4d2c\" (UID: \"bcedd47a-6389-43d4-9765-b0a34bfe88f6\") " pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.611986 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.611944 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:46.979102 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:46.979076 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddc6f7dd5-p4d2c"] Nov 02 20:19:46.980969 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:19:46.980942 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcedd47a_6389_43d4_9765_b0a34bfe88f6.slice/crio-a809b9580ab9879dbbca399b7995352f2d72c435cd0114bc5ed01b8500b72556 WatchSource:0}: Error finding container a809b9580ab9879dbbca399b7995352f2d72c435cd0114bc5ed01b8500b72556: Status 404 returned error can't find the container with id a809b9580ab9879dbbca399b7995352f2d72c435cd0114bc5ed01b8500b72556 Nov 02 20:19:47.825091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:47.825053 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" event={"ID":"bcedd47a-6389-43d4-9765-b0a34bfe88f6","Type":"ContainerStarted","Data":"a809b9580ab9879dbbca399b7995352f2d72c435cd0114bc5ed01b8500b72556"} Nov 02 20:19:48.833865 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:48.833821 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" event={"ID":"bcedd47a-6389-43d4-9765-b0a34bfe88f6","Type":"ContainerStarted","Data":"05f04f87905c15ba183c1a0c068d791b87722c6ae912896e93ee2b159a4750d0"} Nov 02 20:19:48.834282 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:48.833914 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:19:48.943352 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:48.943280 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" podStartSLOduration=2.082798883 podStartE2EDuration="2.943266071s" totalImagesPullingTime="860.467188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:48.941056839 +0000 UTC m=+3094.889712681" watchObservedRunningTime="2026-11-02 20:19:48.943266071 +0000 UTC m=+3094.891921910" Nov 02 20:19:59.843030 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:19:59.842952 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ddc6f7dd5-p4d2c" Nov 02 20:20:08.042427 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:20:08.042393 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h"] Nov 02 20:20:08.088101 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:20:08.088063 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-281fe165-66c4h"] Nov 02 20:20:08.182679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:20:08.182645 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d190c1-32e8-4b22-9b4e-0eedcba467d9" path="/var/lib/kubelet/pods/b4d190c1-32e8-4b22-9b4e-0eedcba467d9/volumes" Nov 02 20:20:14.751099 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:20:14.751061 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:20:19.389689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:20:19.389652 4966 scope.go:122] "RemoveContainer" containerID="48edfc0b3e4173adba0a484302af063aa2491ff0d431b9e07976e9ccdb0b7e0c" Nov 02 20:20:19.399750 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:20:19.399727 4966 scope.go:122] "RemoveContainer" containerID="c7d907f3034418c7e14096b4c84832412d68b201c20e3b9d7cabf9d684f4a81f" Nov 02 20:20:19.408694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:20:19.408676 4966 scope.go:122] "RemoveContainer" containerID="388b6c15d0a74dc74fc4d7dc6bbedeb9181f52fbd63617ee0b307b8b51195e8c" Nov 02 20:20:44.752605 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:20:44.752560 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:21:02.260487 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.260443 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm"] Nov 02 20:21:02.264458 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.264433 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371482 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371441 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371482 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371484 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371735 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371664 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371828 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371800 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371877 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371866 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371934 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539d4970-b4c7-4ecf-ab7c-be947313ce04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371989 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371965 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.371989 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.371985 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.372059 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.372017 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/539d4970-b4c7-4ecf-ab7c-be947313ce04-kube-api-access-8mfrx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.456213 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.456177 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm"] Nov 02 20:21:02.473530 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473502 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.473688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473560 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.473688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473586 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539d4970-b4c7-4ecf-ab7c-be947313ce04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.473688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473604 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.473688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473624 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.473688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473641 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/539d4970-b4c7-4ecf-ab7c-be947313ce04-kube-api-access-8mfrx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.473688 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473678 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.474050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473707 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.474050 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.473774 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.474169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.474041 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.474169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.474085 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.474279 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.474184 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.474627 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.474606 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.474727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.474657 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539d4970-b4c7-4ecf-ab7c-be947313ce04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.476333 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.476286 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.476615 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.476595 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.534802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.534742 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539d4970-b4c7-4ecf-ab7c-be947313ce04-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.534996 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.534974 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/539d4970-b4c7-4ecf-ab7c-be947313ce04-kube-api-access-8mfrx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tkxm\" (UID: \"539d4970-b4c7-4ecf-ab7c-be947313ce04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:02.589972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:02.589937 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:03.007313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.007268 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm"] Nov 02 20:21:03.045089 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:21:03.045057 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539d4970_b4c7_4ecf_ab7c_be947313ce04.slice/crio-87b38542db86d8012c17a43e76c74578095e3da2bed4aafcd9785043ffaa4314 WatchSource:0}: Error finding container 87b38542db86d8012c17a43e76c74578095e3da2bed4aafcd9785043ffaa4314: Status 404 returned error can't find the container with id 87b38542db86d8012c17a43e76c74578095e3da2bed4aafcd9785043ffaa4314 Nov 02 20:21:03.047209 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.047177 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:21:03.047315 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.047244 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:21:03.047315 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.047270 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:21:03.320682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.320591 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" event={"ID":"539d4970-b4c7-4ecf-ab7c-be947313ce04","Type":"ContainerStarted","Data":"37c3419cf69645130b99ceae23e164820b6c4ee5b98a12dc623a000cb636d403"} Nov 02 20:21:03.320682 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.320654 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" event={"ID":"539d4970-b4c7-4ecf-ab7c-be947313ce04","Type":"ContainerStarted","Data":"87b38542db86d8012c17a43e76c74578095e3da2bed4aafcd9785043ffaa4314"} Nov 02 20:21:03.432875 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.432814 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podStartSLOduration=1.432798086 podStartE2EDuration="1.432798086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:03.429514014 +0000 UTC m=+3169.378169852" watchObservedRunningTime="2026-11-02 20:21:03.432798086 +0000 UTC m=+3169.381453933" Nov 02 20:21:03.590239 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:03.590153 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:04.054396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:04.054348 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw"] Nov 02 20:21:04.095506 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:04.095463 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037b901d-579cw"] Nov 02 20:21:04.183330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:04.183277 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84274e96-6147-4b01-b9f6-4202e2c05641" path="/var/lib/kubelet/pods/84274e96-6147-4b01-b9f6-4202e2c05641/volumes" Nov 02 20:21:04.591896 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:04.591857 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 02 20:21:04.592356 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:04.591924 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 02 20:21:05.062899 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:05.062850 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg"] Nov 02 20:21:05.093712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:05.093663 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bede2ba-d8xzg"] Nov 02 20:21:05.590695 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:05.590658 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 02 20:21:05.590911 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:05.590726 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 02 20:21:06.182178 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:06.182138 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa22eabd-1266-4a41-b3ab-e26b9c47722c" path="/var/lib/kubelet/pods/aa22eabd-1266-4a41-b3ab-e26b9c47722c/volumes" Nov 02 20:21:06.590603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:06.590554 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 02 20:21:06.590800 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:06.590633 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 02 20:21:07.590570 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:07.590531 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 02 20:21:07.591098 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:07.590595 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 02 20:21:08.590875 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:08.590822 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 02 20:21:08.591417 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:08.590893 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 02 20:21:09.590614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:09.590569 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 02 20:21:09.590851 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:09.590642 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 02 20:21:10.591108 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:10.591060 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 02 20:21:10.591586 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:10.591136 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 02 20:21:12.631469 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:12.631427 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:21:12.631972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:12.631505 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:21:13.455053 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:13.455018 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:13.455346 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:13.455322 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:13.456125 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:13.456103 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" Nov 02 20:21:14.751915 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:21:14.751879 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:21:19.446170 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:19.446136 4966 scope.go:122] "RemoveContainer" containerID="0edbdac4538de16819e51829a566c198e0fc3389ca2631806ccd7ccf2725c55b" Nov 02 20:21:19.456236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:19.456212 4966 scope.go:122] "RemoveContainer" containerID="0b82f4c553c987f2b052d4f4fcc4260bd655f9c708f6cc952990c729d4acf5b9" Nov 02 20:21:19.465066 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:19.465042 4966 scope.go:122] "RemoveContainer" containerID="ffa41b58073908eaa64dbf6e36e74000eabc882e6e0454b42de1f2e5cdd5984d" Nov 02 20:21:19.475524 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:19.475502 4966 scope.go:122] "RemoveContainer" containerID="b0f87465fe0c834fe76bbc9e1b45d0f63566d48bf1dcc55eed88c383532ba4e0" Nov 02 20:21:19.487117 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:21:19.487090 4966 scope.go:122] "RemoveContainer" containerID="11337a81eb390fa7d917ad70acee0e5a0f1b7d6f2c7d0bfb154dc575aeeb96f2" Nov 02 20:21:44.753796 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:21:44.753734 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:22:02.684224 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.684186 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww"] Nov 02 20:22:02.688000 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.687983 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767192 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767147 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767405 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767198 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767405 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767375 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767530 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767413 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tstxj\" (UniqueName: \"kubernetes.io/projected/895f69ba-a25f-44c4-ab3a-be99dddd93f1-kube-api-access-tstxj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767530 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767439 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767551 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767579 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767594 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.767774 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.767734 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.785076 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.785043 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww"] Nov 02 20:22:02.868770 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.868715 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.868770 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.868770 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.868798 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.868870 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.868910 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.868938 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869011 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869049 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869047 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstxj\" (UniqueName: \"kubernetes.io/projected/895f69ba-a25f-44c4-ab3a-be99dddd93f1-kube-api-access-tstxj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869083 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869292 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869429 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869471 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869508 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.869655 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.869603 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.871264 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.871241 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.871615 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.871596 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.921233 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.921205 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f69ba-a25f-44c4-ab3a-be99dddd93f1-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:02.921744 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:02.921718 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstxj\" (UniqueName: \"kubernetes.io/projected/895f69ba-a25f-44c4-ab3a-be99dddd93f1-kube-api-access-tstxj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-98mww\" (UID: \"895f69ba-a25f-44c4-ab3a-be99dddd93f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:03.001369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.001340 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:03.414736 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.414707 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww"] Nov 02 20:22:03.415467 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:22:03.415438 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895f69ba_a25f_44c4_ab3a_be99dddd93f1.slice/crio-16a13ce413c02d2ebe516ce15db13d67b9d5e94a3ce74852c0dbc3855aa020d6 WatchSource:0}: Error finding container 16a13ce413c02d2ebe516ce15db13d67b9d5e94a3ce74852c0dbc3855aa020d6: Status 404 returned error can't find the container with id 16a13ce413c02d2ebe516ce15db13d67b9d5e94a3ce74852c0dbc3855aa020d6 Nov 02 20:22:03.417645 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.417612 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:03.417713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.417680 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:03.417713 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.417710 4966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:03.742062 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.742027 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" event={"ID":"895f69ba-a25f-44c4-ab3a-be99dddd93f1","Type":"ContainerStarted","Data":"9aa880b9339934b1b2547e8510b30344469ffaa8889db296f11010daa6f8e21d"} Nov 02 20:22:03.742522 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.742091 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" event={"ID":"895f69ba-a25f-44c4-ab3a-be99dddd93f1","Type":"ContainerStarted","Data":"16a13ce413c02d2ebe516ce15db13d67b9d5e94a3ce74852c0dbc3855aa020d6"} Nov 02 20:22:03.853111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:03.853046 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podStartSLOduration=1.8530309219999999 podStartE2EDuration="1.853030922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:03.849980724 +0000 UTC m=+3229.798636563" watchObservedRunningTime="2026-11-02 20:22:03.853030922 +0000 UTC m=+3229.801686758" Nov 02 20:22:04.001842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:04.001743 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:04.003452 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:04.003425 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 02 20:22:04.003617 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:04.003484 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 02 20:22:05.002018 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:05.001977 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 02 20:22:05.002444 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:05.002048 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 02 20:22:06.002363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:06.002323 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 02 20:22:06.002785 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:06.002409 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 02 20:22:07.002017 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:07.001976 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 02 20:22:07.002267 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:07.002045 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 02 20:22:08.002045 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.002004 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 02 20:22:08.002519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.002089 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 02 20:22:08.062669 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.062630 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb"] Nov 02 20:22:08.104561 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.104517 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4"] Nov 02 20:22:08.106825 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.106792 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e2070a4-nq5lb"] Nov 02 20:22:08.108748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.108683 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5ec09-rmtr4"] Nov 02 20:22:08.184573 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.184540 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5790175d-42f0-4e57-abcd-fb125f7b359c" path="/var/lib/kubelet/pods/5790175d-42f0-4e57-abcd-fb125f7b359c/volumes" Nov 02 20:22:08.185091 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:08.185073 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55edd7b-db3a-40b9-b23c-157349b3cc36" path="/var/lib/kubelet/pods/e55edd7b-db3a-40b9-b23c-157349b3cc36/volumes" Nov 02 20:22:09.002495 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:09.002456 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 02 20:22:09.002906 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:09.002525 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 02 20:22:10.002755 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:10.002708 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 02 20:22:10.003168 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:10.002780 4966 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 02 20:22:11.775942 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:11.775894 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:11.776411 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:11.776310 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:11.777133 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:11.777107 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" Nov 02 20:22:14.753083 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:22:14.753034 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:22:19.535562 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:19.535529 4966 scope.go:122] "RemoveContainer" containerID="c14d8cd97997c20da5dc3baf442949b53047360feb2f148d5478917b9d58b6ed" Nov 02 20:22:19.545505 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:19.545478 4966 scope.go:122] "RemoveContainer" containerID="e86d9150b665b4d2f889093754b394960cd1f6848f330c33c92af2d7fbecae07" Nov 02 20:22:19.554792 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:19.554772 4966 scope.go:122] "RemoveContainer" containerID="09db9f46674482a9d4ae616335ed87f17f9dec3207351eb3b4fc76793c5a8eec" Nov 02 20:22:19.563842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:19.563820 4966 scope.go:122] "RemoveContainer" containerID="ebadc432d5c4e551b3cabc7c9102a3c9fd094efdb89475444d323feb1d296e15" Nov 02 20:22:19.573153 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:19.573127 4966 scope.go:122] "RemoveContainer" containerID="c598c9ea8b8dce92539fdd1f71b1b76384fdf5e978682278e5b7e7750b59f9ce" Nov 02 20:22:19.581891 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:19.581868 4966 scope.go:122] "RemoveContainer" containerID="e7d010d367b39f1a0f4bc114fd947defe0bb1e18b184c1267ee7b271e5044a87" Nov 02 20:22:27.817553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:27.817509 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:22:27.818103 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:27.817586 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:44.501426 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:44.501384 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:22:44.502189 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:22:44.502151 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:44.763673 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:22:44.763541 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:23:14.750864 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:23:14.750830 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:23:15.161868 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:15.161776 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:23:15.182270 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:15.182235 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:23:15.183119 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:15.183096 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:23:15.202866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:15.202831 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:23:19.624571 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:19.624530 4966 scope.go:122] "RemoveContainer" containerID="6db63625c1d5bde8d581f197c98200a8cf467149d9bb5c06e37ef43ae13b5970" Nov 02 20:23:19.634470 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:19.634448 4966 scope.go:122] "RemoveContainer" containerID="0fb0a816663cea4054663df6871fb8225a8eaf3f51416c519434bcad0716569e" Nov 02 20:23:19.643250 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:19.643232 4966 scope.go:122] "RemoveContainer" containerID="cceb8a99af76b6c59b51d90f2f3e7706f1116bcb4ea44cb0752373b3d416f1ad" Nov 02 20:23:19.651767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:19.651742 4966 scope.go:122] "RemoveContainer" containerID="9480149b27128365157159b34cc53864d66bac7bdd791e1cb79a5024636d6a4b" Nov 02 20:23:19.660587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:19.660563 4966 scope.go:122] "RemoveContainer" containerID="2d212ae271ce0fd81316261bbd44cde242b27a8dc30b7a58217ead036d788951" Nov 02 20:23:19.669006 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:19.668985 4966 scope.go:122] "RemoveContainer" containerID="7d4253b0285c2703035f13c6d403cda2a56ffd3294a3d9abe9e701df78cc750d" Nov 02 20:23:23.049082 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:23.049041 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt"] Nov 02 20:23:23.094891 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:23.094858 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7478163d-w4sxt"] Nov 02 20:23:24.191547 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:24.191503 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3a7500-a873-4ca7-a6dc-32dd12b7d049" path="/var/lib/kubelet/pods/4c3a7500-a873-4ca7-a6dc-32dd12b7d049/volumes" Nov 02 20:23:41.042739 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:41.042703 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd"] Nov 02 20:23:41.088853 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:41.088816 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5d1f7b-mqsrd"] Nov 02 20:23:42.182692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:23:42.182653 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc31d5c-545a-4047-9e1d-e2c562d7808b" path="/var/lib/kubelet/pods/0cc31d5c-545a-4047-9e1d-e2c562d7808b/volumes" Nov 02 20:23:44.753501 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:23:44.753391 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:24:12.820519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:12.820472 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-98mww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:24:12.821069 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:12.820562 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-98mww" podUID="895f69ba-a25f-44c4-ab3a-be99dddd93f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:24:14.751904 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:24:14.751868 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:24:19.710036 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:19.709999 4966 scope.go:122] "RemoveContainer" containerID="ca9e271df50b8df737a92dc5e02abe0add64d454f4afee02de6e90eaa9b7d3f0" Nov 02 20:24:19.719387 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:19.719362 4966 scope.go:122] "RemoveContainer" containerID="0b724c2640367655d3450096c41e6a7946368c1378caa290334340377a5af5ea" Nov 02 20:24:19.728177 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:19.728151 4966 scope.go:122] "RemoveContainer" containerID="f6be1e8533fb66114f6e47a55337997f79da4672beb3f2b685e3a83852cfe704" Nov 02 20:24:19.739552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:19.739524 4966 scope.go:122] "RemoveContainer" containerID="e526de837f87e10d8f52ca3078c4e62108a50d3fa782b327ee698b35a2ebf243" Nov 02 20:24:44.508325 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:44.507408 4966 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:24:44.508325 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:24:44.507493 4966 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tkxm" podUID="539d4970-b4c7-4ecf-ab7c-be947313ce04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:24:44.817508 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:24:44.817417 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:25:14.752215 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:25:14.752173 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:25:19.775383 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:25:19.775352 4966 scope.go:122] "RemoveContainer" containerID="7033f18f7dd5184bd6d88f4064b138a3584ea592afa4acca10d1aac919928689" Nov 02 20:25:19.784089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:25:19.784068 4966 scope.go:122] "RemoveContainer" containerID="1bd1b61bc9085c7603d243a37bb8a3c458667fb475a54d9d9e7853d2e49f0e8d" Nov 02 20:25:44.752648 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:25:44.752608 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:26:14.751646 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:26:14.751609 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:26:41.173494 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.173453 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r"] Nov 02 20:26:41.177336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.177278 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.259712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.259680 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-tmp-dir\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.259882 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.259728 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kws9c\" (UniqueName: \"kubernetes.io/projected/c2c98c97-34dc-4498-827f-7b6fae9101ec-kube-api-access-kws9c\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.259882 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.259815 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c98c97-34dc-4498-827f-7b6fae9101ec-tls-certs\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.259962 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.259906 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-dshm\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.259962 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.259956 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-home\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.260063 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.260042 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-model-cache\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.274911 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.274873 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9ff720\"/\"kube-root-ca.crt\"" Nov 02 20:26:41.274911 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.274873 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9ff720\"/\"openshift-service-ca.crt\"" Nov 02 20:26:41.275119 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.275076 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9ff720\"/\"default-dockercfg-2fqn9\"" Nov 02 20:26:41.275162 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.275130 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9ff720\"/\"simulator-single-main-9ff720-kserve-self-signed-certs\"" Nov 02 20:26:41.277402 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.277383 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r"] Nov 02 20:26:41.361397 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361359 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-dshm\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.361603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361440 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-home\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.361603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361493 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-model-cache\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.361603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361538 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-tmp-dir\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.361603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361571 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kws9c\" (UniqueName: \"kubernetes.io/projected/c2c98c97-34dc-4498-827f-7b6fae9101ec-kube-api-access-kws9c\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.361603 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361597 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c98c97-34dc-4498-827f-7b6fae9101ec-tls-certs\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.361937 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361913 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-home\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.362002 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361948 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-tmp-dir\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.362002 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.361986 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-model-cache\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.364195 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.364129 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-dshm\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.364409 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.364386 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c98c97-34dc-4498-827f-7b6fae9101ec-tls-certs\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.415086 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.415045 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kws9c\" (UniqueName: \"kubernetes.io/projected/c2c98c97-34dc-4498-827f-7b6fae9101ec-kube-api-access-kws9c\") pod \"simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.490032 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.489985 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:41.867465 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:41.867432 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r"] Nov 02 20:26:41.869445 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:26:41.869417 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2c98c97_34dc_4498_827f_7b6fae9101ec.slice/crio-0d791c81be3d1a5f6fbfc341b4b6dbac30942fc4b38bbd9aa52a3d2c25bad426 WatchSource:0}: Error finding container 0d791c81be3d1a5f6fbfc341b4b6dbac30942fc4b38bbd9aa52a3d2c25bad426: Status 404 returned error can't find the container with id 0d791c81be3d1a5f6fbfc341b4b6dbac30942fc4b38bbd9aa52a3d2c25bad426 Nov 02 20:26:42.479711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:42.479670 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" event={"ID":"c2c98c97-34dc-4498-827f-7b6fae9101ec","Type":"ContainerStarted","Data":"a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d"} Nov 02 20:26:42.480183 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:42.479721 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" event={"ID":"c2c98c97-34dc-4498-827f-7b6fae9101ec","Type":"ContainerStarted","Data":"0d791c81be3d1a5f6fbfc341b4b6dbac30942fc4b38bbd9aa52a3d2c25bad426"} Nov 02 20:26:42.490881 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:42.490855 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:42.588824 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:42.588768 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" podStartSLOduration=1.588752954 podStartE2EDuration="1.588752954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:42.586332371 +0000 UTC m=+3508.534988204" watchObservedRunningTime="2026-11-02 20:26:42.588752954 +0000 UTC m=+3508.537408792" Nov 02 20:26:44.753084 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:26:44.753040 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:26:51.490819 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:51.490780 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:51.503537 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:51.503507 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:26:51.544602 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:26:51.544577 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:27:14.752061 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:27:14.752024 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:27:41.942225 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:41.942182 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r"] Nov 02 20:27:44.754898 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:27:44.754855 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:27:56.963076 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:56.963004 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" podUID="c2c98c97-34dc-4498-827f-7b6fae9101ec" containerName="main" containerID="cri-o://a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d" gracePeriod=45 Nov 02 20:27:57.224899 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.224875 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:27:57.418830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.418795 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c98c97-34dc-4498-827f-7b6fae9101ec-kube-api-access-kws9c\" (UniqueName: \"kubernetes.io/projected/c2c98c97-34dc-4498-827f-7b6fae9101ec-kube-api-access-kws9c\") pod \"c2c98c97-34dc-4498-827f-7b6fae9101ec\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " Nov 02 20:27:57.418830 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.418845 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c98c97-34dc-4498-827f-7b6fae9101ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c98c97-34dc-4498-827f-7b6fae9101ec-tls-certs\") pod \"c2c98c97-34dc-4498-827f-7b6fae9101ec\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " Nov 02 20:27:57.419114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.418913 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-tmp-dir\") pod \"c2c98c97-34dc-4498-827f-7b6fae9101ec\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " Nov 02 20:27:57.419114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.418933 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-home\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-home\") pod \"c2c98c97-34dc-4498-827f-7b6fae9101ec\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " Nov 02 20:27:57.419114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.418993 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-dshm\") pod \"c2c98c97-34dc-4498-827f-7b6fae9101ec\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " Nov 02 20:27:57.419114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.419016 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-model-cache\") pod \"c2c98c97-34dc-4498-827f-7b6fae9101ec\" (UID: \"c2c98c97-34dc-4498-827f-7b6fae9101ec\") " Nov 02 20:27:57.419362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.419215 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-tmp-dir" pod "c2c98c97-34dc-4498-827f-7b6fae9101ec" (UID: "c2c98c97-34dc-4498-827f-7b6fae9101ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:57.419362 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.419233 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-home" pod "c2c98c97-34dc-4498-827f-7b6fae9101ec" (UID: "c2c98c97-34dc-4498-827f-7b6fae9101ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:57.419467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.419439 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:57.419514 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.419455 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-model-cache" pod "c2c98c97-34dc-4498-827f-7b6fae9101ec" (UID: "c2c98c97-34dc-4498-827f-7b6fae9101ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:57.419514 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.419465 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:57.421221 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.421189 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-dshm" pod "c2c98c97-34dc-4498-827f-7b6fae9101ec" (UID: "c2c98c97-34dc-4498-827f-7b6fae9101ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:27:57.421439 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.421414 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c98c97-34dc-4498-827f-7b6fae9101ec-kube-api-access-kws9c" pod "c2c98c97-34dc-4498-827f-7b6fae9101ec" (UID: "c2c98c97-34dc-4498-827f-7b6fae9101ec"). InnerVolumeSpecName "kube-api-access-kws9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:27:57.421552 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.421534 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c98c97-34dc-4498-827f-7b6fae9101ec-tls-certs" pod "c2c98c97-34dc-4498-827f-7b6fae9101ec" (UID: "c2c98c97-34dc-4498-827f-7b6fae9101ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:27:57.520538 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.520445 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:57.520538 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.520478 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c98c97-34dc-4498-827f-7b6fae9101ec-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:57.520538 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.520489 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kws9c\" (UniqueName: \"kubernetes.io/projected/c2c98c97-34dc-4498-827f-7b6fae9101ec-kube-api-access-kws9c\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:57.520538 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.520500 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c98c97-34dc-4498-827f-7b6fae9101ec-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:57.919835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.919742 4966 generic.go:356] "Generic (PLEG): container finished" podID="c2c98c97-34dc-4498-827f-7b6fae9101ec" containerID="a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d" exitCode=0 Nov 02 20:27:57.919835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.919786 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" Nov 02 20:27:57.919835 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.919807 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" event={"ID":"c2c98c97-34dc-4498-827f-7b6fae9101ec","Type":"ContainerDied","Data":"a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d"} Nov 02 20:27:57.920120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.919848 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r" event={"ID":"c2c98c97-34dc-4498-827f-7b6fae9101ec","Type":"ContainerDied","Data":"0d791c81be3d1a5f6fbfc341b4b6dbac30942fc4b38bbd9aa52a3d2c25bad426"} Nov 02 20:27:57.920120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.919869 4966 scope.go:122] "RemoveContainer" containerID="a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d" Nov 02 20:27:57.930114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.930087 4966 scope.go:122] "RemoveContainer" containerID="a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d" Nov 02 20:27:57.930418 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:27:57.930394 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d\": container with ID starting with a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d not found: ID does not exist" containerID="a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d" Nov 02 20:27:57.930493 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:57.930427 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d"} err="failed to get container status \"a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d\": rpc error: code = NotFound desc = could not find container \"a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d\": container with ID starting with a112b609e283b6a6a8c28800fc26a98192a009038462998f9b83fe0a855f048d not found: ID does not exist" Nov 02 20:27:58.071887 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:58.071842 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r"] Nov 02 20:27:58.117764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:58.117720 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-9ff720/simulator-single-main-9ff720-kserve-6b7bc66f68-p2p9r"] Nov 02 20:27:58.183672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:27:58.183590 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c98c97-34dc-4498-827f-7b6fae9101ec" path="/var/lib/kubelet/pods/c2c98c97-34dc-4498-827f-7b6fae9101ec/volumes" Nov 02 20:28:14.750968 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:28:14.750928 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:28:15.210808 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:15.210730 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:28:15.223829 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:15.223805 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:28:15.233095 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:15.233076 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:28:15.245534 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:15.245513 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:28:27.421554 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.421512 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h"] Nov 02 20:28:27.422071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.421746 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" podUID="482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" containerName="manager" containerID="cri-o://ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10" gracePeriod=10 Nov 02 20:28:27.673845 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.673774 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 20:28:27.801010 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.800974 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-kube-api-access-kvwxm\" (UniqueName: \"kubernetes.io/projected/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-kube-api-access-kvwxm\") pod \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " Nov 02 20:28:27.801217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.801099 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-extensions-socket-volume\") pod \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\" (UID: \"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed\") " Nov 02 20:28:27.801517 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.801492 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-extensions-socket-volume" pod "482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" (UID: "482a3710-b7e5-4d30-bc84-c37fdb5cf8ed"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:27.803288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.803262 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-kube-api-access-kvwxm" pod "482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" (UID: "482a3710-b7e5-4d30-bc84-c37fdb5cf8ed"). InnerVolumeSpecName "kube-api-access-kvwxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:27.901971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.901931 4966 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-extensions-socket-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:27.901971 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:27.901970 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvwxm\" (UniqueName: \"kubernetes.io/projected/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed-kube-api-access-kvwxm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:28.086860 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.086819 4966 generic.go:356] "Generic (PLEG): container finished" podID="482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" containerID="ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10" exitCode=0 Nov 02 20:28:28.087081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.086868 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" Nov 02 20:28:28.087081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.086914 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" event={"ID":"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed","Type":"ContainerDied","Data":"ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10"} Nov 02 20:28:28.087081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.086976 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h" event={"ID":"482a3710-b7e5-4d30-bc84-c37fdb5cf8ed","Type":"ContainerDied","Data":"510dcd86cfab67228cc88337f9ec16175a09fba32c24f90d5d237ba2ab1c1050"} Nov 02 20:28:28.087081 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.087005 4966 scope.go:122] "RemoveContainer" containerID="ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10" Nov 02 20:28:28.098977 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.098955 4966 scope.go:122] "RemoveContainer" containerID="ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10" Nov 02 20:28:28.099240 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:28:28.099219 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10\": container with ID starting with ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10 not found: ID does not exist" containerID="ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10" Nov 02 20:28:28.099328 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.099249 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10"} err="failed to get container status \"ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10\": rpc error: code = NotFound desc = could not find container \"ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10\": container with ID starting with ef457ffe5a05f0b516be1e362f7fc670cb2455dc768a1e8cd0a81c95b6a67b10 not found: ID does not exist" Nov 02 20:28:28.241078 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.241044 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h"] Nov 02 20:28:28.286863 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:28.286829 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wgs9h"] Nov 02 20:28:30.182567 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:28:30.182530 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" path="/var/lib/kubelet/pods/482a3710-b7e5-4d30-bc84-c37fdb5cf8ed/volumes" Nov 02 20:28:44.751841 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:28:44.751801 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:29:14.751733 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:29:14.751691 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:29:31.103564 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.103528 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp"] Nov 02 20:29:31.104006 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.103934 4966 cpu_manager.go:414] "containerMap: removing container" podUID="482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" containerName="manager" Nov 02 20:29:31.104006 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.103945 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" containerName="manager" Nov 02 20:29:31.104006 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.103961 4966 cpu_manager.go:414] "containerMap: removing container" podUID="c2c98c97-34dc-4498-827f-7b6fae9101ec" containerName="main" Nov 02 20:29:31.104006 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.103966 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c98c97-34dc-4498-827f-7b6fae9101ec" containerName="main" Nov 02 20:29:31.104192 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.104070 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="482a3710-b7e5-4d30-bc84-c37fdb5cf8ed" containerName="manager" Nov 02 20:29:31.104192 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.104081 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c98c97-34dc-4498-827f-7b6fae9101ec" containerName="main" Nov 02 20:29:31.108471 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.108415 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.205264 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.205227 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4gc5k\"" Nov 02 20:29:31.207633 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.207605 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp"] Nov 02 20:29:31.207767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.207629 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11b4daa-eb53-4acf-b7e6-2a22b154d939-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-t2nvp\" (UID: \"e11b4daa-eb53-4acf-b7e6-2a22b154d939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.207767 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.207736 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jnj\" (UniqueName: \"kubernetes.io/projected/e11b4daa-eb53-4acf-b7e6-2a22b154d939-kube-api-access-r9jnj\") pod \"kuadrant-operator-controller-manager-678d87766f-t2nvp\" (UID: \"e11b4daa-eb53-4acf-b7e6-2a22b154d939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.309019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.308973 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11b4daa-eb53-4acf-b7e6-2a22b154d939-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-t2nvp\" (UID: \"e11b4daa-eb53-4acf-b7e6-2a22b154d939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.309248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.309081 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jnj\" (UniqueName: \"kubernetes.io/projected/e11b4daa-eb53-4acf-b7e6-2a22b154d939-kube-api-access-r9jnj\") pod \"kuadrant-operator-controller-manager-678d87766f-t2nvp\" (UID: \"e11b4daa-eb53-4acf-b7e6-2a22b154d939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.309543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.309522 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11b4daa-eb53-4acf-b7e6-2a22b154d939-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-t2nvp\" (UID: \"e11b4daa-eb53-4acf-b7e6-2a22b154d939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.365563 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.365495 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jnj\" (UniqueName: \"kubernetes.io/projected/e11b4daa-eb53-4acf-b7e6-2a22b154d939-kube-api-access-r9jnj\") pod \"kuadrant-operator-controller-manager-678d87766f-t2nvp\" (UID: \"e11b4daa-eb53-4acf-b7e6-2a22b154d939\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.422267 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.422234 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:31.883928 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:31.883894 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp"] Nov 02 20:29:31.885197 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:29:31.885171 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11b4daa_eb53_4acf_b7e6_2a22b154d939.slice/crio-d64213979d62e5d609a8b0e4fd73ffedbe4816ef8278c7083e1de9eed42fb6f6 WatchSource:0}: Error finding container d64213979d62e5d609a8b0e4fd73ffedbe4816ef8278c7083e1de9eed42fb6f6: Status 404 returned error can't find the container with id d64213979d62e5d609a8b0e4fd73ffedbe4816ef8278c7083e1de9eed42fb6f6 Nov 02 20:29:32.436738 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:32.436694 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" event={"ID":"e11b4daa-eb53-4acf-b7e6-2a22b154d939","Type":"ContainerStarted","Data":"c6e2fc03987f9c704720cc735e8291a82cbf5c903e83c3667bcb4e4ee5ae6c4b"} Nov 02 20:29:32.437184 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:32.436759 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" event={"ID":"e11b4daa-eb53-4acf-b7e6-2a22b154d939","Type":"ContainerStarted","Data":"d64213979d62e5d609a8b0e4fd73ffedbe4816ef8278c7083e1de9eed42fb6f6"} Nov 02 20:29:32.437184 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:32.436910 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:32.546167 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:32.546111 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" podStartSLOduration=1.5460966379999999 podStartE2EDuration="1.546096638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:32.544491927 +0000 UTC m=+3678.493147760" watchObservedRunningTime="2026-11-02 20:29:32.546096638 +0000 UTC m=+3678.494752476" Nov 02 20:29:43.446034 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:29:43.446003 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t2nvp" Nov 02 20:29:44.751591 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:29:44.751554 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:30:00.152225 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.152186 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp"] Nov 02 20:30:00.156022 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.155992 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.172439 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.172414 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9853e08e-fba0-4e18-9bc5-1327b7a48174-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-r82gp\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.172576 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.172448 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wj7w\" (UniqueName: \"kubernetes.io/projected/9853e08e-fba0-4e18-9bc5-1327b7a48174-kube-api-access-9wj7w\") pod \"maas-api-key-cleanup-29894190-r82gp\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.252806 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.252770 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qqnp8\"" Nov 02 20:30:00.252806 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.252776 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:30:00.253010 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.252783 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:30:00.255344 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.255315 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp"] Nov 02 20:30:00.273503 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.273475 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9853e08e-fba0-4e18-9bc5-1327b7a48174-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-r82gp\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.273627 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.273508 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wj7w\" (UniqueName: \"kubernetes.io/projected/9853e08e-fba0-4e18-9bc5-1327b7a48174-kube-api-access-9wj7w\") pod \"maas-api-key-cleanup-29894190-r82gp\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.274219 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.274197 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9853e08e-fba0-4e18-9bc5-1327b7a48174-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-r82gp\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.325776 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.325745 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wj7w\" (UniqueName: \"kubernetes.io/projected/9853e08e-fba0-4e18-9bc5-1327b7a48174-kube-api-access-9wj7w\") pod \"maas-api-key-cleanup-29894190-r82gp\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.469363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.469333 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:00.831260 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:00.831230 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp"] Nov 02 20:30:00.833532 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:30:00.833492 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9853e08e_fba0_4e18_9bc5_1327b7a48174.slice/crio-8fc752f544d2b29a7b27be2a7953284f8c052db0fece546ae75b6adc30354c28 WatchSource:0}: Error finding container 8fc752f544d2b29a7b27be2a7953284f8c052db0fece546ae75b6adc30354c28: Status 404 returned error can't find the container with id 8fc752f544d2b29a7b27be2a7953284f8c052db0fece546ae75b6adc30354c28 Nov 02 20:30:01.614790 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:01.614748 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" event={"ID":"9853e08e-fba0-4e18-9bc5-1327b7a48174","Type":"ContainerStarted","Data":"4e5bb88afaf2c9151cbfcf3c61a0811952fdab101f631213b29840404a861b01"} Nov 02 20:30:01.615214 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:01.614809 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" event={"ID":"9853e08e-fba0-4e18-9bc5-1327b7a48174","Type":"ContainerStarted","Data":"8fc752f544d2b29a7b27be2a7953284f8c052db0fece546ae75b6adc30354c28"} Nov 02 20:30:01.721260 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:01.721199 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" podStartSLOduration=1.721184084 podStartE2EDuration="1.721184084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:01.719011989 +0000 UTC m=+3707.667667827" watchObservedRunningTime="2026-11-02 20:30:01.721184084 +0000 UTC m=+3707.669839923" Nov 02 20:30:02.622463 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:02.622427 4966 generic.go:356] "Generic (PLEG): container finished" podID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerID="4e5bb88afaf2c9151cbfcf3c61a0811952fdab101f631213b29840404a861b01" exitCode=60 Nov 02 20:30:02.622892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:02.622470 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" event={"ID":"9853e08e-fba0-4e18-9bc5-1327b7a48174","Type":"ContainerDied","Data":"4e5bb88afaf2c9151cbfcf3c61a0811952fdab101f631213b29840404a861b01"} Nov 02 20:30:02.622892 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:02.622751 4966 scope.go:122] "RemoveContainer" containerID="4e5bb88afaf2c9151cbfcf3c61a0811952fdab101f631213b29840404a861b01" Nov 02 20:30:03.630466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:03.630431 4966 generic.go:356] "Generic (PLEG): container finished" podID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerID="f8b683d9444c50fc99fce1c8159b1d6c24ae70288c6381daa1e8c1bb84c976d3" exitCode=60 Nov 02 20:30:03.630895 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:03.630494 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" event={"ID":"9853e08e-fba0-4e18-9bc5-1327b7a48174","Type":"ContainerDied","Data":"f8b683d9444c50fc99fce1c8159b1d6c24ae70288c6381daa1e8c1bb84c976d3"} Nov 02 20:30:03.630895 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:03.630541 4966 scope.go:122] "RemoveContainer" containerID="4e5bb88afaf2c9151cbfcf3c61a0811952fdab101f631213b29840404a861b01" Nov 02 20:30:03.630895 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:03.630742 4966 scope.go:122] "RemoveContainer" containerID="f8b683d9444c50fc99fce1c8159b1d6c24ae70288c6381daa1e8c1bb84c976d3" Nov 02 20:30:03.631039 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:30:03.631009 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-r82gp_odh-ai-gateway-infra(9853e08e-fba0-4e18-9bc5-1327b7a48174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" Nov 02 20:30:04.637222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:04.637190 4966 scope.go:122] "RemoveContainer" containerID="f8b683d9444c50fc99fce1c8159b1d6c24ae70288c6381daa1e8c1bb84c976d3" Nov 02 20:30:04.637656 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:30:04.637453 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-r82gp_odh-ai-gateway-infra(9853e08e-fba0-4e18-9bc5-1327b7a48174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" Nov 02 20:30:14.760087 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:30:14.760027 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:30:18.176161 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:18.176123 4966 scope.go:122] "RemoveContainer" containerID="f8b683d9444c50fc99fce1c8159b1d6c24ae70288c6381daa1e8c1bb84c976d3" Nov 02 20:30:18.740604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:18.740561 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" event={"ID":"9853e08e-fba0-4e18-9bc5-1327b7a48174","Type":"ContainerStarted","Data":"cc9dd520835ffce0e685bf9365118eb004e2125e31ec19ff69626d05029f9da3"} Nov 02 20:30:19.748041 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:19.748007 4966 generic.go:356] "Generic (PLEG): container finished" podID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerID="cc9dd520835ffce0e685bf9365118eb004e2125e31ec19ff69626d05029f9da3" exitCode=60 Nov 02 20:30:19.748556 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:19.748082 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" event={"ID":"9853e08e-fba0-4e18-9bc5-1327b7a48174","Type":"ContainerDied","Data":"cc9dd520835ffce0e685bf9365118eb004e2125e31ec19ff69626d05029f9da3"} Nov 02 20:30:19.748556 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:19.748146 4966 scope.go:122] "RemoveContainer" containerID="f8b683d9444c50fc99fce1c8159b1d6c24ae70288c6381daa1e8c1bb84c976d3" Nov 02 20:30:19.748556 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:19.748503 4966 scope.go:122] "RemoveContainer" containerID="cc9dd520835ffce0e685bf9365118eb004e2125e31ec19ff69626d05029f9da3" Nov 02 20:30:19.748776 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:30:19.748752 4966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-r82gp_odh-ai-gateway-infra(9853e08e-fba0-4e18-9bc5-1327b7a48174)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" Nov 02 20:30:19.890650 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:19.890611 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp"] Nov 02 20:30:20.899803 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:20.899778 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:20.974668 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:20.974627 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9853e08e-fba0-4e18-9bc5-1327b7a48174-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9853e08e-fba0-4e18-9bc5-1327b7a48174-maas-api-service-ca\") pod \"9853e08e-fba0-4e18-9bc5-1327b7a48174\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " Nov 02 20:30:20.974877 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:20.974722 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9853e08e-fba0-4e18-9bc5-1327b7a48174-kube-api-access-9wj7w\" (UniqueName: \"kubernetes.io/projected/9853e08e-fba0-4e18-9bc5-1327b7a48174-kube-api-access-9wj7w\") pod \"9853e08e-fba0-4e18-9bc5-1327b7a48174\" (UID: \"9853e08e-fba0-4e18-9bc5-1327b7a48174\") " Nov 02 20:30:20.975030 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:20.975006 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9853e08e-fba0-4e18-9bc5-1327b7a48174-maas-api-service-ca" pod "9853e08e-fba0-4e18-9bc5-1327b7a48174" (UID: "9853e08e-fba0-4e18-9bc5-1327b7a48174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:20.977074 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:20.977041 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9853e08e-fba0-4e18-9bc5-1327b7a48174-kube-api-access-9wj7w" pod "9853e08e-fba0-4e18-9bc5-1327b7a48174" (UID: "9853e08e-fba0-4e18-9bc5-1327b7a48174"). InnerVolumeSpecName "kube-api-access-9wj7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:21.076388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:21.076285 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9853e08e-fba0-4e18-9bc5-1327b7a48174-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.076388 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:21.076343 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wj7w\" (UniqueName: \"kubernetes.io/projected/9853e08e-fba0-4e18-9bc5-1327b7a48174-kube-api-access-9wj7w\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.760974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:21.760941 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" Nov 02 20:30:21.761176 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:21.760970 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp" event={"ID":"9853e08e-fba0-4e18-9bc5-1327b7a48174","Type":"ContainerDied","Data":"8fc752f544d2b29a7b27be2a7953284f8c052db0fece546ae75b6adc30354c28"} Nov 02 20:30:21.761176 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:21.761031 4966 scope.go:122] "RemoveContainer" containerID="cc9dd520835ffce0e685bf9365118eb004e2125e31ec19ff69626d05029f9da3" Nov 02 20:30:21.912727 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:21.912689 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp"] Nov 02 20:30:21.958874 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:21.958838 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-r82gp"] Nov 02 20:30:22.183168 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:30:22.183091 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" path="/var/lib/kubelet/pods/9853e08e-fba0-4e18-9bc5-1327b7a48174/volumes" Nov 02 20:30:44.752062 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:30:44.752019 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:31:06.255631 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.255584 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md"] Nov 02 20:31:06.256245 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.256221 4966 cpu_manager.go:414] "containerMap: removing container" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:31:06.256370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.256248 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:31:06.256370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.256265 4966 cpu_manager.go:414] "containerMap: removing container" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:31:06.256370 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.256273 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:31:06.256618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.256455 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:31:06.256618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.256479 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:31:06.256618 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.256488 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:31:06.259945 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.259922 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.310569 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.310426 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a52dfa\"/\"kube-root-ca.crt\"" Nov 02 20:31:06.314988 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.314958 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-model-cache\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.315154 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.314996 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-home\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.315154 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.315135 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231e2d48-840d-4743-a264-0a4e16d6c15d-tls-certs\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.315281 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.315175 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-tmp-dir\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.315365 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.315270 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-dshm\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.315425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.315357 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cnt\" (UniqueName: \"kubernetes.io/projected/231e2d48-840d-4743-a264-0a4e16d6c15d-kube-api-access-76cnt\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.334410 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.334374 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a52dfa\"/\"openshift-service-ca.crt\"" Nov 02 20:31:06.356756 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.356714 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a52dfa\"/\"default-dockercfg-njqdj\"" Nov 02 20:31:06.356945 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.356769 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a52dfa\"/\"simulator-single-main-a52dfa-kserve-self-signed-certs\"" Nov 02 20:31:06.359073 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.359046 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md"] Nov 02 20:31:06.416466 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.416430 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231e2d48-840d-4743-a264-0a4e16d6c15d-tls-certs\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.416679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.416484 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-tmp-dir\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.416679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.416520 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-dshm\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.416679 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.416547 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cnt\" (UniqueName: \"kubernetes.io/projected/231e2d48-840d-4743-a264-0a4e16d6c15d-kube-api-access-76cnt\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.416876 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.416802 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-model-cache\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.416876 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.416838 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-home\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.417398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.417372 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-home\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.417691 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.417670 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-tmp-dir\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.418779 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.418749 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-model-cache\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.421151 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.421056 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231e2d48-840d-4743-a264-0a4e16d6c15d-tls-certs\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.422217 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.422187 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-dshm\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.470025 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.469988 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cnt\" (UniqueName: \"kubernetes.io/projected/231e2d48-840d-4743-a264-0a4e16d6c15d-kube-api-access-76cnt\") pod \"simulator-single-main-a52dfa-kserve-5b55d8d47-m28md\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.571821 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.571735 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:06.947032 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:06.947005 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md"] Nov 02 20:31:06.948777 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:31:06.948747 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod231e2d48_840d_4743_a264_0a4e16d6c15d.slice/crio-b423d03e42fd69cbbc9275cec91de16b59acd8e4eb680660656c6f10b3fc7483 WatchSource:0}: Error finding container b423d03e42fd69cbbc9275cec91de16b59acd8e4eb680660656c6f10b3fc7483: Status 404 returned error can't find the container with id b423d03e42fd69cbbc9275cec91de16b59acd8e4eb680660656c6f10b3fc7483 Nov 02 20:31:07.024491 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:07.024455 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" event={"ID":"231e2d48-840d-4743-a264-0a4e16d6c15d","Type":"ContainerStarted","Data":"b423d03e42fd69cbbc9275cec91de16b59acd8e4eb680660656c6f10b3fc7483"} Nov 02 20:31:08.032277 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:08.032236 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" event={"ID":"231e2d48-840d-4743-a264-0a4e16d6c15d","Type":"ContainerStarted","Data":"df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639"} Nov 02 20:31:08.138510 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:08.138450 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" podStartSLOduration=2.13843482 podStartE2EDuration="2.13843482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:08.137390767 +0000 UTC m=+3774.086046606" watchObservedRunningTime="2026-11-02 20:31:08.13843482 +0000 UTC m=+3774.087090661" Nov 02 20:31:08.572079 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:08.572048 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:14.751172 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:31:14.751126 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:31:16.572963 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:16.572915 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:16.587734 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:16.587706 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:17.098004 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:31:17.097973 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:31:44.752793 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:31:44.752755 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:32:14.752223 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:32:14.752183 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:32:17.437467 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:17.437430 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md"] Nov 02 20:32:32.458030 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.457968 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" podUID="231e2d48-840d-4743-a264-0a4e16d6c15d" containerName="main" containerID="cri-o://df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639" gracePeriod=45 Nov 02 20:32:32.711612 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.711552 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:32:32.778820 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.778787 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231e2d48-840d-4743-a264-0a4e16d6c15d-tls-certs\" (UniqueName: \"kubernetes.io/secret/231e2d48-840d-4743-a264-0a4e16d6c15d-tls-certs\") pod \"231e2d48-840d-4743-a264-0a4e16d6c15d\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " Nov 02 20:32:32.779019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.778861 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-home\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-home\") pod \"231e2d48-840d-4743-a264-0a4e16d6c15d\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " Nov 02 20:32:32.779019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.778893 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-tmp-dir\") pod \"231e2d48-840d-4743-a264-0a4e16d6c15d\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " Nov 02 20:32:32.779019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.778918 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-model-cache\") pod \"231e2d48-840d-4743-a264-0a4e16d6c15d\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " Nov 02 20:32:32.779019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.778942 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231e2d48-840d-4743-a264-0a4e16d6c15d-kube-api-access-76cnt\" (UniqueName: \"kubernetes.io/projected/231e2d48-840d-4743-a264-0a4e16d6c15d-kube-api-access-76cnt\") pod \"231e2d48-840d-4743-a264-0a4e16d6c15d\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " Nov 02 20:32:32.779227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.779080 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-dshm\") pod \"231e2d48-840d-4743-a264-0a4e16d6c15d\" (UID: \"231e2d48-840d-4743-a264-0a4e16d6c15d\") " Nov 02 20:32:32.779227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.779172 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-home" pod "231e2d48-840d-4743-a264-0a4e16d6c15d" (UID: "231e2d48-840d-4743-a264-0a4e16d6c15d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:32.779227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.779191 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-tmp-dir" pod "231e2d48-840d-4743-a264-0a4e16d6c15d" (UID: "231e2d48-840d-4743-a264-0a4e16d6c15d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:32.779227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.779204 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-model-cache" pod "231e2d48-840d-4743-a264-0a4e16d6c15d" (UID: "231e2d48-840d-4743-a264-0a4e16d6c15d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:32.779472 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.779424 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:32.779472 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.779436 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:32.779472 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.779445 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:32.781289 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.781260 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231e2d48-840d-4743-a264-0a4e16d6c15d-kube-api-access-76cnt" pod "231e2d48-840d-4743-a264-0a4e16d6c15d" (UID: "231e2d48-840d-4743-a264-0a4e16d6c15d"). InnerVolumeSpecName "kube-api-access-76cnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:32.781424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.781261 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231e2d48-840d-4743-a264-0a4e16d6c15d-tls-certs" pod "231e2d48-840d-4743-a264-0a4e16d6c15d" (UID: "231e2d48-840d-4743-a264-0a4e16d6c15d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:32.781424 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.781328 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-dshm" pod "231e2d48-840d-4743-a264-0a4e16d6c15d" (UID: "231e2d48-840d-4743-a264-0a4e16d6c15d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:32.880712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.880668 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cnt\" (UniqueName: \"kubernetes.io/projected/231e2d48-840d-4743-a264-0a4e16d6c15d-kube-api-access-76cnt\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:32.880712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.880700 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/231e2d48-840d-4743-a264-0a4e16d6c15d-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:32.880712 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:32.880710 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/231e2d48-840d-4743-a264-0a4e16d6c15d-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:33.533121 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.533084 4966 generic.go:356] "Generic (PLEG): container finished" podID="231e2d48-840d-4743-a264-0a4e16d6c15d" containerID="df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639" exitCode=0 Nov 02 20:32:33.533611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.533129 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" Nov 02 20:32:33.533611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.533172 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" event={"ID":"231e2d48-840d-4743-a264-0a4e16d6c15d","Type":"ContainerDied","Data":"df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639"} Nov 02 20:32:33.533611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.533215 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md" event={"ID":"231e2d48-840d-4743-a264-0a4e16d6c15d","Type":"ContainerDied","Data":"b423d03e42fd69cbbc9275cec91de16b59acd8e4eb680660656c6f10b3fc7483"} Nov 02 20:32:33.533611 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.533231 4966 scope.go:122] "RemoveContainer" containerID="df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639" Nov 02 20:32:33.543415 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.543393 4966 scope.go:122] "RemoveContainer" containerID="df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639" Nov 02 20:32:33.543752 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:32:33.543730 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639\": container with ID starting with df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639 not found: ID does not exist" containerID="df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639" Nov 02 20:32:33.543807 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.543762 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639"} err="failed to get container status \"df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639\": rpc error: code = NotFound desc = could not find container \"df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639\": container with ID starting with df6b2e7a2c9a0e2e2333e0e4ca4a27bcef1e0ee6650da778a297d88867a25639 not found: ID does not exist" Nov 02 20:32:33.690775 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.690743 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md"] Nov 02 20:32:33.692863 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:33.692833 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a52dfa/simulator-single-main-a52dfa-kserve-5b55d8d47-m28md"] Nov 02 20:32:34.182131 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:32:34.182097 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231e2d48-840d-4743-a264-0a4e16d6c15d" path="/var/lib/kubelet/pods/231e2d48-840d-4743-a264-0a4e16d6c15d/volumes" Nov 02 20:32:44.751854 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:32:44.751816 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:33:14.751364 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:33:14.751323 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:33:15.248363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:15.248335 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:33:15.261711 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:15.261684 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:33:15.271995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:15.271973 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:33:15.285662 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:15.285643 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:33:17.118246 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.118215 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8"] Nov 02 20:33:17.118686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.118630 4966 cpu_manager.go:414] "containerMap: removing container" podUID="231e2d48-840d-4743-a264-0a4e16d6c15d" containerName="main" Nov 02 20:33:17.118686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.118641 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e2d48-840d-4743-a264-0a4e16d6c15d" containerName="main" Nov 02 20:33:17.118686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.118660 4966 cpu_manager.go:414] "containerMap: removing container" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:33:17.118686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.118667 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9853e08e-fba0-4e18-9bc5-1327b7a48174" containerName="cleanup" Nov 02 20:33:17.118834 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.118748 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="231e2d48-840d-4743-a264-0a4e16d6c15d" containerName="main" Nov 02 20:33:17.122021 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.121997 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.192111 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.192072 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-dshm\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.192339 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.192132 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049c7d64-b65f-4cb0-aeac-035b63d8269e-tls-certs\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.192339 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.192312 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-home\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.192470 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.192366 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-tmp-dir\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.192515 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.192490 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-model-cache\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.192574 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.192549 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svs2l\" (UniqueName: \"kubernetes.io/projected/049c7d64-b65f-4cb0-aeac-035b63d8269e-kube-api-access-svs2l\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.219285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.219246 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 20:33:17.219285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.219263 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 20:33:17.219285 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.219282 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xwhqg\"" Nov 02 20:33:17.219597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.219476 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7265e7a4-kserve-self-signed-certs\"" Nov 02 20:33:17.221521 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.221500 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8"] Nov 02 20:33:17.293476 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.293443 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-dshm\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.293515 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049c7d64-b65f-4cb0-aeac-035b63d8269e-tls-certs\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.293846 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-home\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.293917 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-tmp-dir\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.293985 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-model-cache\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.294039 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svs2l\" (UniqueName: \"kubernetes.io/projected/049c7d64-b65f-4cb0-aeac-035b63d8269e-kube-api-access-svs2l\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.294293 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-home\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294745 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.294446 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-tmp-dir\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.294745 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.294482 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-model-cache\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.295932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.295911 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-dshm\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.296312 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.296281 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049c7d64-b65f-4cb0-aeac-035b63d8269e-tls-certs\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.347407 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.347370 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svs2l\" (UniqueName: \"kubernetes.io/projected/049c7d64-b65f-4cb0-aeac-035b63d8269e-kube-api-access-svs2l\") pod \"e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.434904 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.434817 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:17.816593 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:17.816570 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8"] Nov 02 20:33:17.817881 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:33:17.817855 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod049c7d64_b65f_4cb0_aeac_035b63d8269e.slice/crio-01d229160dcfe659abc54e626e68d77ad12b8e2c8facf9320e5d8bbfe3b2aff4 WatchSource:0}: Error finding container 01d229160dcfe659abc54e626e68d77ad12b8e2c8facf9320e5d8bbfe3b2aff4: Status 404 returned error can't find the container with id 01d229160dcfe659abc54e626e68d77ad12b8e2c8facf9320e5d8bbfe3b2aff4 Nov 02 20:33:18.796594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:18.796558 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" event={"ID":"049c7d64-b65f-4cb0-aeac-035b63d8269e","Type":"ContainerStarted","Data":"e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f"} Nov 02 20:33:18.797011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:18.796602 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" event={"ID":"049c7d64-b65f-4cb0-aeac-035b63d8269e","Type":"ContainerStarted","Data":"01d229160dcfe659abc54e626e68d77ad12b8e2c8facf9320e5d8bbfe3b2aff4"} Nov 02 20:33:18.903493 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:18.903442 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" podStartSLOduration=1.9034265719999999 podStartE2EDuration="1.903426572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:18.902072301 +0000 UTC m=+3904.850728141" watchObservedRunningTime="2026-11-02 20:33:18.903426572 +0000 UTC m=+3904.852082410" Nov 02 20:33:19.435130 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:19.435093 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:27.435215 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:27.435135 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:27.448186 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:27.448161 4966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:27.864311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:33:27.864260 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:33:44.752442 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:33:44.752400 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:34:14.767517 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:34:14.767477 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:34:25.309443 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:25.309410 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8"] Nov 02 20:34:40.329595 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.329544 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" podUID="049c7d64-b65f-4cb0-aeac-035b63d8269e" containerName="main" containerID="cri-o://e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f" gracePeriod=45 Nov 02 20:34:40.586094 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.586040 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:34:40.706700 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.706662 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-tmp-dir\") pod \"049c7d64-b65f-4cb0-aeac-035b63d8269e\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " Nov 02 20:34:40.706898 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.706779 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-dshm\") pod \"049c7d64-b65f-4cb0-aeac-035b63d8269e\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " Nov 02 20:34:40.706898 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.706820 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-model-cache\") pod \"049c7d64-b65f-4cb0-aeac-035b63d8269e\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " Nov 02 20:34:40.706898 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.706852 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049c7d64-b65f-4cb0-aeac-035b63d8269e-tls-certs\" (UniqueName: \"kubernetes.io/secret/049c7d64-b65f-4cb0-aeac-035b63d8269e-tls-certs\") pod \"049c7d64-b65f-4cb0-aeac-035b63d8269e\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " Nov 02 20:34:40.707065 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.706903 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049c7d64-b65f-4cb0-aeac-035b63d8269e-kube-api-access-svs2l\" (UniqueName: \"kubernetes.io/projected/049c7d64-b65f-4cb0-aeac-035b63d8269e-kube-api-access-svs2l\") pod \"049c7d64-b65f-4cb0-aeac-035b63d8269e\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " Nov 02 20:34:40.707065 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.706940 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-home\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-home\") pod \"049c7d64-b65f-4cb0-aeac-035b63d8269e\" (UID: \"049c7d64-b65f-4cb0-aeac-035b63d8269e\") " Nov 02 20:34:40.707065 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.706988 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-tmp-dir" pod "049c7d64-b65f-4cb0-aeac-035b63d8269e" (UID: "049c7d64-b65f-4cb0-aeac-035b63d8269e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:40.707363 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.707337 4966 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:40.707565 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.707435 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-model-cache" pod "049c7d64-b65f-4cb0-aeac-035b63d8269e" (UID: "049c7d64-b65f-4cb0-aeac-035b63d8269e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:40.707630 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.707566 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-home" pod "049c7d64-b65f-4cb0-aeac-035b63d8269e" (UID: "049c7d64-b65f-4cb0-aeac-035b63d8269e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:40.709426 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.709397 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049c7d64-b65f-4cb0-aeac-035b63d8269e-tls-certs" pod "049c7d64-b65f-4cb0-aeac-035b63d8269e" (UID: "049c7d64-b65f-4cb0-aeac-035b63d8269e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:40.709741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.709716 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c7d64-b65f-4cb0-aeac-035b63d8269e-kube-api-access-svs2l" pod "049c7d64-b65f-4cb0-aeac-035b63d8269e" (UID: "049c7d64-b65f-4cb0-aeac-035b63d8269e"). InnerVolumeSpecName "kube-api-access-svs2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:40.709930 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.709904 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-dshm" pod "049c7d64-b65f-4cb0-aeac-035b63d8269e" (UID: "049c7d64-b65f-4cb0-aeac-035b63d8269e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:40.808235 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.808199 4966 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:40.808235 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.808232 4966 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:40.808235 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.808242 4966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049c7d64-b65f-4cb0-aeac-035b63d8269e-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:40.808587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.808251 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svs2l\" (UniqueName: \"kubernetes.io/projected/049c7d64-b65f-4cb0-aeac-035b63d8269e-kube-api-access-svs2l\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:40.808587 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:40.808262 4966 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049c7d64-b65f-4cb0-aeac-035b63d8269e-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:41.356726 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.356694 4966 generic.go:356] "Generic (PLEG): container finished" podID="049c7d64-b65f-4cb0-aeac-035b63d8269e" containerID="e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f" exitCode=0 Nov 02 20:34:41.357222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.356746 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" Nov 02 20:34:41.357222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.356775 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" event={"ID":"049c7d64-b65f-4cb0-aeac-035b63d8269e","Type":"ContainerDied","Data":"e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f"} Nov 02 20:34:41.357222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.356826 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8" event={"ID":"049c7d64-b65f-4cb0-aeac-035b63d8269e","Type":"ContainerDied","Data":"01d229160dcfe659abc54e626e68d77ad12b8e2c8facf9320e5d8bbfe3b2aff4"} Nov 02 20:34:41.357222 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.356849 4966 scope.go:122] "RemoveContainer" containerID="e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f" Nov 02 20:34:41.367594 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.367574 4966 scope.go:122] "RemoveContainer" containerID="e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f" Nov 02 20:34:41.367881 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:34:41.367861 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f\": container with ID starting with e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f not found: ID does not exist" containerID="e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f" Nov 02 20:34:41.367952 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.367889 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f"} err="failed to get container status \"e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f\": rpc error: code = NotFound desc = could not find container \"e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f\": container with ID starting with e68c6ddc8e870347140228ca8beb27d9d017d8086f241df38157f7346db4157f not found: ID does not exist" Nov 02 20:34:41.508116 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.508080 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8"] Nov 02 20:34:41.553793 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:41.553750 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7265e7a4-kserve-65f4858449-cb8j8"] Nov 02 20:34:42.182057 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:34:42.182026 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049c7d64-b65f-4cb0-aeac-035b63d8269e" path="/var/lib/kubelet/pods/049c7d64-b65f-4cb0-aeac-035b63d8269e/volumes" Nov 02 20:34:44.755453 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:34:44.755412 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:35:14.751906 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:35:14.751862 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:35:44.751680 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:35:44.751634 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:35:55.047321 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.047260 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74"] Nov 02 20:35:55.047910 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.047891 4966 cpu_manager.go:414] "containerMap: removing container" podUID="049c7d64-b65f-4cb0-aeac-035b63d8269e" containerName="main" Nov 02 20:35:55.047910 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.047909 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049c7d64-b65f-4cb0-aeac-035b63d8269e" containerName="main" Nov 02 20:35:55.048034 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.048013 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="049c7d64-b65f-4cb0-aeac-035b63d8269e" containerName="main" Nov 02 20:35:55.051223 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.051202 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.083886 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.083855 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.084034 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.083904 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.084087 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.084056 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbqf\" (UniqueName: \"kubernetes.io/projected/ff73da5b-153b-4a6b-9c27-af7b07737dbd-kube-api-access-rjbqf\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.102071 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.102039 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:55.105278 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.105253 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74"] Nov 02 20:35:55.147765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.147733 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:35:55.147897 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.147794 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrp5q\"" Nov 02 20:35:55.148047 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.148029 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:35:55.184872 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.184846 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbqf\" (UniqueName: \"kubernetes.io/projected/ff73da5b-153b-4a6b-9c27-af7b07737dbd-kube-api-access-rjbqf\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.185002 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.184901 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.185002 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.184923 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.185641 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.185623 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.187368 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.187352 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.238734 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.238703 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbqf\" (UniqueName: \"kubernetes.io/projected/ff73da5b-153b-4a6b-9c27-af7b07737dbd-kube-api-access-rjbqf\") pod \"maas-api-revoke-keys-e2e-ait-5b04055d-zvt74\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.364442 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.364355 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:55.729391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.729365 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74"] Nov 02 20:35:55.731524 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:35:55.731496 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff73da5b_153b_4a6b_9c27_af7b07737dbd.slice/crio-9d623b871c8f1cf75ee4d132bdd7e9444bbf0dcdca0e96c65ff7f28e304d2b76 WatchSource:0}: Error finding container 9d623b871c8f1cf75ee4d132bdd7e9444bbf0dcdca0e96c65ff7f28e304d2b76: Status 404 returned error can't find the container with id 9d623b871c8f1cf75ee4d132bdd7e9444bbf0dcdca0e96c65ff7f28e304d2b76 Nov 02 20:35:55.788657 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:55.788621 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" event={"ID":"ff73da5b-153b-4a6b-9c27-af7b07737dbd","Type":"ContainerStarted","Data":"9d623b871c8f1cf75ee4d132bdd7e9444bbf0dcdca0e96c65ff7f28e304d2b76"} Nov 02 20:35:56.796200 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:56.796165 4966 generic.go:356] "Generic (PLEG): container finished" podID="ff73da5b-153b-4a6b-9c27-af7b07737dbd" containerID="9187d0e727c70d0225e207101785eae5151e36a5ab12a5b8bf01e1d3088cbb89" exitCode=0 Nov 02 20:35:56.796660 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:56.796247 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" event={"ID":"ff73da5b-153b-4a6b-9c27-af7b07737dbd","Type":"ContainerDied","Data":"9187d0e727c70d0225e207101785eae5151e36a5ab12a5b8bf01e1d3088cbb89"} Nov 02 20:35:57.934391 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:57.934364 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:58.011277 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.011243 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-xks-ca\") pod \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " Nov 02 20:35:58.011465 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.011292 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff73da5b-153b-4a6b-9c27-af7b07737dbd-kube-api-access-rjbqf\" (UniqueName: \"kubernetes.io/projected/ff73da5b-153b-4a6b-9c27-af7b07737dbd-kube-api-access-rjbqf\") pod \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " Nov 02 20:35:58.011465 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.011416 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-openshift-ca\") pod \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\" (UID: \"ff73da5b-153b-4a6b-9c27-af7b07737dbd\") " Nov 02 20:35:58.011799 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.011772 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-openshift-ca" pod "ff73da5b-153b-4a6b-9c27-af7b07737dbd" (UID: "ff73da5b-153b-4a6b-9c27-af7b07737dbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:58.013336 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.013292 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-xks-ca" pod "ff73da5b-153b-4a6b-9c27-af7b07737dbd" (UID: "ff73da5b-153b-4a6b-9c27-af7b07737dbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:58.013425 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.013374 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff73da5b-153b-4a6b-9c27-af7b07737dbd-kube-api-access-rjbqf" pod "ff73da5b-153b-4a6b-9c27-af7b07737dbd" (UID: "ff73da5b-153b-4a6b-9c27-af7b07737dbd"). InnerVolumeSpecName "kube-api-access-rjbqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:58.112553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.112461 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:58.112553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.112503 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbqf\" (UniqueName: \"kubernetes.io/projected/ff73da5b-153b-4a6b-9c27-af7b07737dbd-kube-api-access-rjbqf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:58.112553 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.112520 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff73da5b-153b-4a6b-9c27-af7b07737dbd-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:58.809019 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.808982 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" Nov 02 20:35:58.809241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.809016 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74" event={"ID":"ff73da5b-153b-4a6b-9c27-af7b07737dbd","Type":"ContainerDied","Data":"9d623b871c8f1cf75ee4d132bdd7e9444bbf0dcdca0e96c65ff7f28e304d2b76"} Nov 02 20:35:58.809241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:35:58.809068 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d623b871c8f1cf75ee4d132bdd7e9444bbf0dcdca0e96c65ff7f28e304d2b76" Nov 02 20:36:13.347571 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:13.347536 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74"] Nov 02 20:36:13.393555 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:13.393524 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b04055d-zvt74"] Nov 02 20:36:14.183243 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:14.183209 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff73da5b-153b-4a6b-9c27-af7b07737dbd" path="/var/lib/kubelet/pods/ff73da5b-153b-4a6b-9c27-af7b07737dbd/volumes" Nov 02 20:36:14.754335 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:36:14.753273 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:36:25.982229 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:25.982141 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26"] Nov 02 20:36:25.982731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:25.982663 4966 cpu_manager.go:414] "containerMap: removing container" podUID="ff73da5b-153b-4a6b-9c27-af7b07737dbd" containerName="revoke-keys" Nov 02 20:36:25.982731 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:25.982679 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff73da5b-153b-4a6b-9c27-af7b07737dbd" containerName="revoke-keys" Nov 02 20:36:25.982816 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:25.982748 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff73da5b-153b-4a6b-9c27-af7b07737dbd" containerName="revoke-keys" Nov 02 20:36:25.985629 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:25.985613 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.080123 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.080088 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-cfa6c5-dockercfg-lpnvq\"" Nov 02 20:36:26.080123 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.080110 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-cfa6c5\"" Nov 02 20:36:26.082509 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.082484 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26"] Nov 02 20:36:26.091765 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.091733 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.091878 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.091773 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdjp\" (UniqueName: \"kubernetes.io/projected/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-kube-api-access-nzdjp\") pod \"payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.193062 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.193025 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdjp\" (UniqueName: \"kubernetes.io/projected/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-kube-api-access-nzdjp\") pod \"payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.193241 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.193178 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.193936 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.193913 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.246099 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.246016 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdjp\" (UniqueName: \"kubernetes.io/projected/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-kube-api-access-nzdjp\") pod \"payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.298997 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.298952 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.663124 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.663097 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26"] Nov 02 20:36:26.665232 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:36:26.665199 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388da9db_18a3_4d6e_97dc_e9cf2f1f93b0.slice/crio-9b1e7c29f6523c0a3314ade5c7f9830947423b811aafead8e181554df1de8753 WatchSource:0}: Error finding container 9b1e7c29f6523c0a3314ade5c7f9830947423b811aafead8e181554df1de8753: Status 404 returned error can't find the container with id 9b1e7c29f6523c0a3314ade5c7f9830947423b811aafead8e181554df1de8753 Nov 02 20:36:26.976558 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.976515 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" event={"ID":"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0","Type":"ContainerStarted","Data":"468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31"} Nov 02 20:36:26.976751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.976591 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:26.976751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:26.976616 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" event={"ID":"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0","Type":"ContainerStarted","Data":"9b1e7c29f6523c0a3314ade5c7f9830947423b811aafead8e181554df1de8753"} Nov 02 20:36:27.081932 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:27.081867 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" podStartSLOduration=2.081845866 podStartE2EDuration="2.081845866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:27.081597538 +0000 UTC m=+4093.030253372" watchObservedRunningTime="2026-11-02 20:36:27.081845866 +0000 UTC m=+4093.030501708" Nov 02 20:36:37.989085 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:36:37.989051 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:36:44.752738 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:36:44.752699 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:37:14.753816 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:37:14.753769 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:37:44.752086 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:37:44.752050 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:38:10.023801 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.023713 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6"] Nov 02 20:38:10.027313 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.027281 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.081156 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.081119 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6"] Nov 02 20:38:10.114745 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.114706 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.114944 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.114761 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cq9b\" (UniqueName: \"kubernetes.io/projected/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-kube-api-access-4cq9b\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.114944 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.114859 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.124330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.124285 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:38:10.124330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.124292 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:38:10.124551 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.124396 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:38:10.124551 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.124470 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrp5q\"" Nov 02 20:38:10.216147 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.216108 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.216147 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.216146 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq9b\" (UniqueName: \"kubernetes.io/projected/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-kube-api-access-4cq9b\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.216437 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.216228 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.216927 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.216906 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.218632 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.218611 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.269316 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.269273 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq9b\" (UniqueName: \"kubernetes.io/projected/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-kube-api-access-4cq9b\") pod \"maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.346750 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.346670 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:10.713719 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:10.713691 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6"] Nov 02 20:38:10.715709 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:38:10.715683 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b615ab9_b3e9_444d_a9b5_9b5d1dc2e32c.slice/crio-a97571c121d4893c3737e7a822f9fa9eebba7b72c3568c66c4be835072a188d1 WatchSource:0}: Error finding container a97571c121d4893c3737e7a822f9fa9eebba7b72c3568c66c4be835072a188d1: Status 404 returned error can't find the container with id a97571c121d4893c3737e7a822f9fa9eebba7b72c3568c66c4be835072a188d1 Nov 02 20:38:11.552048 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:11.552009 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" event={"ID":"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c","Type":"ContainerStarted","Data":"d6e7c1bdaf4c4d65439b2170ea5317b43c0d39a45b8e9fa517c99a7efb4f3c16"} Nov 02 20:38:11.552519 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:11.552067 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" event={"ID":"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c","Type":"ContainerStarted","Data":"a97571c121d4893c3737e7a822f9fa9eebba7b72c3568c66c4be835072a188d1"} Nov 02 20:38:11.657202 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:11.657152 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" podStartSLOduration=2.65713795 podStartE2EDuration="2.65713795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:11.656866019 +0000 UTC m=+4197.605521857" watchObservedRunningTime="2026-11-02 20:38:11.65713795 +0000 UTC m=+4197.605793785" Nov 02 20:38:12.559922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:12.559885 4966 generic.go:356] "Generic (PLEG): container finished" podID="8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" containerID="d6e7c1bdaf4c4d65439b2170ea5317b43c0d39a45b8e9fa517c99a7efb4f3c16" exitCode=0 Nov 02 20:38:12.560416 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:12.559976 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" event={"ID":"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c","Type":"ContainerDied","Data":"d6e7c1bdaf4c4d65439b2170ea5317b43c0d39a45b8e9fa517c99a7efb4f3c16"} Nov 02 20:38:13.711933 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.711904 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:13.854694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.854603 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-kube-api-access-4cq9b\" (UniqueName: \"kubernetes.io/projected/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-kube-api-access-4cq9b\") pod \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " Nov 02 20:38:13.854694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.854659 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-openshift-ca\") pod \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " Nov 02 20:38:13.854901 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.854726 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-xks-ca\") pod \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\" (UID: \"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c\") " Nov 02 20:38:13.855051 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.855030 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-openshift-ca" pod "8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" (UID: "8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:13.856866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.856840 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-kube-api-access-4cq9b" pod "8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" (UID: "8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c"). InnerVolumeSpecName "kube-api-access-4cq9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:13.856866 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.856856 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-xks-ca" pod "8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" (UID: "8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:13.956105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.956066 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:13.956105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.956102 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cq9b\" (UniqueName: \"kubernetes.io/projected/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-kube-api-access-4cq9b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:13.956105 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:13.956114 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:14.574325 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:14.574267 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" Nov 02 20:38:14.574520 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:14.574318 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6" event={"ID":"8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c","Type":"ContainerDied","Data":"a97571c121d4893c3737e7a822f9fa9eebba7b72c3568c66c4be835072a188d1"} Nov 02 20:38:14.574520 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:14.574371 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a97571c121d4893c3737e7a822f9fa9eebba7b72c3568c66c4be835072a188d1" Nov 02 20:38:14.751653 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:38:14.751618 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:38:15.286698 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:15.286670 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:38:15.300958 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:15.300929 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:38:15.314092 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:15.314065 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:38:15.327829 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:15.327801 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:38:21.744708 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:21.744663 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26"] Nov 02 20:38:26.745662 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:26.745587 4966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" podUID="388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" containerName="payload-pre-processing" containerID="cri-o://468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31" gracePeriod=25 Nov 02 20:38:26.997096 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:26.997028 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:38:27.075374 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.075339 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-plugins-config-volume\") pod \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " Nov 02 20:38:27.075575 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.075410 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-kube-api-access-nzdjp\" (UniqueName: \"kubernetes.io/projected/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-kube-api-access-nzdjp\") pod \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\" (UID: \"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0\") " Nov 02 20:38:27.075794 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.075768 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-plugins-config-volume" pod "388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" (UID: "388da9db-18a3-4d6e-97dc-e9cf2f1f93b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:27.077555 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.077535 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-kube-api-access-nzdjp" pod "388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" (UID: "388da9db-18a3-4d6e-97dc-e9cf2f1f93b0"). InnerVolumeSpecName "kube-api-access-nzdjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:27.176145 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.176108 4966 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:27.176145 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.176140 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzdjp\" (UniqueName: \"kubernetes.io/projected/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0-kube-api-access-nzdjp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:27.649119 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.649083 4966 generic.go:356] "Generic (PLEG): container finished" podID="388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" containerID="468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31" exitCode=0 Nov 02 20:38:27.649324 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.649125 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" Nov 02 20:38:27.649324 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.649166 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" event={"ID":"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0","Type":"ContainerDied","Data":"468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31"} Nov 02 20:38:27.649324 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.649222 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26" event={"ID":"388da9db-18a3-4d6e-97dc-e9cf2f1f93b0","Type":"ContainerDied","Data":"9b1e7c29f6523c0a3314ade5c7f9830947423b811aafead8e181554df1de8753"} Nov 02 20:38:27.649324 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.649244 4966 scope.go:122] "RemoveContainer" containerID="468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31" Nov 02 20:38:27.658918 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.658897 4966 scope.go:122] "RemoveContainer" containerID="468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31" Nov 02 20:38:27.659154 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:38:27.659134 4966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31\": container with ID starting with 468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31 not found: ID does not exist" containerID="468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31" Nov 02 20:38:27.659248 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.659163 4966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31"} err="failed to get container status \"468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31\": rpc error: code = NotFound desc = could not find container \"468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31\": container with ID starting with 468837e6b40c18190565e1ecab25e5a1276c6948358bc20e7a5e7a3e9c9e3e31 not found: ID does not exist" Nov 02 20:38:27.800672 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.800639 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26"] Nov 02 20:38:27.846809 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:27.846772 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cfa6c5-7d74586979-qvc26"] Nov 02 20:38:28.182864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:28.182824 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" path="/var/lib/kubelet/pods/388da9db-18a3-4d6e-97dc-e9cf2f1f93b0/volumes" Nov 02 20:38:32.931455 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:32.931412 4966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6"] Nov 02 20:38:32.977396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:32.977359 4966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-cfa6c5-zldd6"] Nov 02 20:38:34.182654 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:38:34.182615 4966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" path="/var/lib/kubelet/pods/8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c/volumes" Nov 02 20:38:44.751862 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:38:44.751822 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:39:06.481002 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.480965 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4"] Nov 02 20:39:06.481624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.481524 4966 cpu_manager.go:414] "containerMap: removing container" podUID="388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" containerName="payload-pre-processing" Nov 02 20:39:06.481624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.481543 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" containerName="payload-pre-processing" Nov 02 20:39:06.481624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.481559 4966 cpu_manager.go:414] "containerMap: removing container" podUID="8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" containerName="revoke-keys" Nov 02 20:39:06.481624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.481568 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" containerName="revoke-keys" Nov 02 20:39:06.481995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.481684 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="388da9db-18a3-4d6e-97dc-e9cf2f1f93b0" containerName="payload-pre-processing" Nov 02 20:39:06.481995 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.481697 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b615ab9-b3e9-444d-a9b5-9b5d1dc2e32c" containerName="revoke-keys" Nov 02 20:39:06.485311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.485284 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.581288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.581243 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:39:06.581288 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.581269 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:39:06.581555 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.581359 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:39:06.581555 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.581396 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrp5q\"" Nov 02 20:39:06.581681 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.581658 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bpg\" (UniqueName: \"kubernetes.io/projected/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-kube-api-access-w6bpg\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.581750 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.581738 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.581883 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.581860 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.583898 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.583878 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4"] Nov 02 20:39:06.683008 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.682975 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bpg\" (UniqueName: \"kubernetes.io/projected/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-kube-api-access-w6bpg\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.683174 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.683047 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.683174 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.683080 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.683780 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.683750 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.685443 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.685423 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.735664 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.735600 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bpg\" (UniqueName: \"kubernetes.io/projected/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-kube-api-access-w6bpg\") pod \"maas-api-revoke-sub-f03629db-67lp4\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:06.798692 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:06.798663 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:07.160518 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:07.160491 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4"] Nov 02 20:39:07.162401 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:39:07.162365 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a2a485_aef1_49cb_b826_b1eef3dfcd70.slice/crio-916e9f9ed254eca6fee7672bda1d1f5c5ae5cacaf782c1c160fe2ee89b495d41 WatchSource:0}: Error finding container 916e9f9ed254eca6fee7672bda1d1f5c5ae5cacaf782c1c160fe2ee89b495d41: Status 404 returned error can't find the container with id 916e9f9ed254eca6fee7672bda1d1f5c5ae5cacaf782c1c160fe2ee89b495d41 Nov 02 20:39:07.919188 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:07.919148 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" event={"ID":"a9a2a485-aef1-49cb-b826-b1eef3dfcd70","Type":"ContainerStarted","Data":"21f62044ed0f11f95fe57fa395583a7ea7c596be25ae79625657601a2e8b5b97"} Nov 02 20:39:07.919614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:07.919204 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" event={"ID":"a9a2a485-aef1-49cb-b826-b1eef3dfcd70","Type":"ContainerStarted","Data":"916e9f9ed254eca6fee7672bda1d1f5c5ae5cacaf782c1c160fe2ee89b495d41"} Nov 02 20:39:08.025070 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:08.025017 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" podStartSLOduration=2.02500038 podStartE2EDuration="2.02500038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:39:08.023731007 +0000 UTC m=+4253.972386846" watchObservedRunningTime="2026-11-02 20:39:08.02500038 +0000 UTC m=+4253.973656219" Nov 02 20:39:08.926901 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:08.926865 4966 generic.go:356] "Generic (PLEG): container finished" podID="a9a2a485-aef1-49cb-b826-b1eef3dfcd70" containerID="21f62044ed0f11f95fe57fa395583a7ea7c596be25ae79625657601a2e8b5b97" exitCode=0 Nov 02 20:39:08.927384 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:08.926958 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" event={"ID":"a9a2a485-aef1-49cb-b826-b1eef3dfcd70","Type":"ContainerDied","Data":"21f62044ed0f11f95fe57fa395583a7ea7c596be25ae79625657601a2e8b5b97"} Nov 02 20:39:10.070409 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.070384 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:10.214251 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.214175 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-openshift-ca\") pod \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " Nov 02 20:39:10.214251 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.214246 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-kube-api-access-w6bpg\" (UniqueName: \"kubernetes.io/projected/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-kube-api-access-w6bpg\") pod \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " Nov 02 20:39:10.214485 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.214283 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-xks-ca\") pod \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\" (UID: \"a9a2a485-aef1-49cb-b826-b1eef3dfcd70\") " Nov 02 20:39:10.214667 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.214639 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-openshift-ca" pod "a9a2a485-aef1-49cb-b826-b1eef3dfcd70" (UID: "a9a2a485-aef1-49cb-b826-b1eef3dfcd70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:10.216492 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.216471 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-xks-ca" pod "a9a2a485-aef1-49cb-b826-b1eef3dfcd70" (UID: "a9a2a485-aef1-49cb-b826-b1eef3dfcd70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:10.216615 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.216592 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-kube-api-access-w6bpg" pod "a9a2a485-aef1-49cb-b826-b1eef3dfcd70" (UID: "a9a2a485-aef1-49cb-b826-b1eef3dfcd70"). InnerVolumeSpecName "kube-api-access-w6bpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:10.315681 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.315647 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:10.315681 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.315676 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bpg\" (UniqueName: \"kubernetes.io/projected/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-kube-api-access-w6bpg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:10.315681 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.315687 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2a485-aef1-49cb-b826-b1eef3dfcd70-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:10.939126 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.939094 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" Nov 02 20:39:10.939354 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.939130 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-67lp4" event={"ID":"a9a2a485-aef1-49cb-b826-b1eef3dfcd70","Type":"ContainerDied","Data":"916e9f9ed254eca6fee7672bda1d1f5c5ae5cacaf782c1c160fe2ee89b495d41"} Nov 02 20:39:10.939354 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:10.939180 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="916e9f9ed254eca6fee7672bda1d1f5c5ae5cacaf782c1c160fe2ee89b495d41" Nov 02 20:39:14.751641 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:39:14.751598 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:39:32.482913 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.482825 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92"] Nov 02 20:39:32.483513 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.483492 4966 cpu_manager.go:414] "containerMap: removing container" podUID="a9a2a485-aef1-49cb-b826-b1eef3dfcd70" containerName="revoke-keys" Nov 02 20:39:32.483608 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.483515 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a2a485-aef1-49cb-b826-b1eef3dfcd70" containerName="revoke-keys" Nov 02 20:39:32.483668 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.483648 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a2a485-aef1-49cb-b826-b1eef3dfcd70" containerName="revoke-keys" Nov 02 20:39:32.486802 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.486782 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.561005 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.560963 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttsr\" (UniqueName: \"kubernetes.io/projected/5cedfbf1-a097-441c-a67a-bd511d6e70c9-kube-api-access-4ttsr\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.561236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.561055 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.561236 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.561175 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.583246 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.583214 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:39:32.583246 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.583222 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:39:32.583468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.583238 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:39:32.583468 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.583393 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrp5q\"" Nov 02 20:39:32.585773 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.585749 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92"] Nov 02 20:39:32.662367 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.662329 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.662542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.662415 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttsr\" (UniqueName: \"kubernetes.io/projected/5cedfbf1-a097-441c-a67a-bd511d6e70c9-kube-api-access-4ttsr\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.662542 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.662443 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.663190 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.663160 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.664655 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.664635 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.715395 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.715364 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttsr\" (UniqueName: \"kubernetes.io/projected/5cedfbf1-a097-441c-a67a-bd511d6e70c9-kube-api-access-4ttsr\") pod \"maas-api-revoke-sub-0165d7d2-2kq92\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:32.799869 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:32.799787 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:33.165675 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:33.165645 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92"] Nov 02 20:39:33.167835 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:39:33.167798 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cedfbf1_a097_441c_a67a_bd511d6e70c9.slice/crio-6fdccd230e429e9d0e8d6534509eac2f532dc8abc87484b2e39db0f94116432b WatchSource:0}: Error finding container 6fdccd230e429e9d0e8d6534509eac2f532dc8abc87484b2e39db0f94116432b: Status 404 returned error can't find the container with id 6fdccd230e429e9d0e8d6534509eac2f532dc8abc87484b2e39db0f94116432b Nov 02 20:39:34.069922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:34.069891 4966 generic.go:356] "Generic (PLEG): container finished" podID="5cedfbf1-a097-441c-a67a-bd511d6e70c9" containerID="4a0746d4402fb0dda30078bee5534fcb8c12301fcdc2465f89af26739c977ee6" exitCode=0 Nov 02 20:39:34.070330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:34.069927 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" event={"ID":"5cedfbf1-a097-441c-a67a-bd511d6e70c9","Type":"ContainerDied","Data":"4a0746d4402fb0dda30078bee5534fcb8c12301fcdc2465f89af26739c977ee6"} Nov 02 20:39:34.070330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:34.069985 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" event={"ID":"5cedfbf1-a097-441c-a67a-bd511d6e70c9","Type":"ContainerStarted","Data":"6fdccd230e429e9d0e8d6534509eac2f532dc8abc87484b2e39db0f94116432b"} Nov 02 20:39:35.211968 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.211939 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:35.279689 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.279656 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cedfbf1-a097-441c-a67a-bd511d6e70c9-kube-api-access-4ttsr\" (UniqueName: \"kubernetes.io/projected/5cedfbf1-a097-441c-a67a-bd511d6e70c9-kube-api-access-4ttsr\") pod \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " Nov 02 20:39:35.279861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.279766 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-openshift-ca\") pod \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " Nov 02 20:39:35.279861 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.279825 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-xks-ca\") pod \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\" (UID: \"5cedfbf1-a097-441c-a67a-bd511d6e70c9\") " Nov 02 20:39:35.280180 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.280153 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-openshift-ca" pod "5cedfbf1-a097-441c-a67a-bd511d6e70c9" (UID: "5cedfbf1-a097-441c-a67a-bd511d6e70c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:35.281911 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.281888 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cedfbf1-a097-441c-a67a-bd511d6e70c9-kube-api-access-4ttsr" pod "5cedfbf1-a097-441c-a67a-bd511d6e70c9" (UID: "5cedfbf1-a097-441c-a67a-bd511d6e70c9"). InnerVolumeSpecName "kube-api-access-4ttsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:35.281911 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.281887 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-xks-ca" pod "5cedfbf1-a097-441c-a67a-bd511d6e70c9" (UID: "5cedfbf1-a097-441c-a67a-bd511d6e70c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:35.380551 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.380458 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ttsr\" (UniqueName: \"kubernetes.io/projected/5cedfbf1-a097-441c-a67a-bd511d6e70c9-kube-api-access-4ttsr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:35.380551 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.380495 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:35.380551 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:35.380519 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cedfbf1-a097-441c-a67a-bd511d6e70c9-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:36.082508 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:36.082471 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" Nov 02 20:39:36.082699 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:36.082507 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2kq92" event={"ID":"5cedfbf1-a097-441c-a67a-bd511d6e70c9","Type":"ContainerDied","Data":"6fdccd230e429e9d0e8d6534509eac2f532dc8abc87484b2e39db0f94116432b"} Nov 02 20:39:36.082699 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:36.082567 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fdccd230e429e9d0e8d6534509eac2f532dc8abc87484b2e39db0f94116432b" Nov 02 20:39:44.159649 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.159611 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7"] Nov 02 20:39:44.160168 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.160034 4966 cpu_manager.go:414] "containerMap: removing container" podUID="5cedfbf1-a097-441c-a67a-bd511d6e70c9" containerName="revoke-keys" Nov 02 20:39:44.160168 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.160046 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cedfbf1-a097-441c-a67a-bd511d6e70c9" containerName="revoke-keys" Nov 02 20:39:44.160748 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.160712 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cedfbf1-a097-441c-a67a-bd511d6e70c9" containerName="revoke-keys" Nov 02 20:39:44.167463 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.165923 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.217907 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.217872 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7"] Nov 02 20:39:44.248766 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.248728 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.248976 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.248897 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.248976 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.248959 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmtm\" (UniqueName: \"kubernetes.io/projected/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-kube-api-access-xkmtm\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.261120 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.261096 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:39:44.261238 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.261115 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:39:44.261238 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.261119 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:39:44.261238 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.261147 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrp5q\"" Nov 02 20:39:44.349673 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.349641 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.349842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.349758 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.349842 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.349803 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmtm\" (UniqueName: \"kubernetes.io/projected/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-kube-api-access-xkmtm\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.350325 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.350278 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.352389 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.352367 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.402601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.402572 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmtm\" (UniqueName: \"kubernetes.io/projected/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-kube-api-access-xkmtm\") pod \"maas-api-revoke-sub-6b19338f-ss4d7\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.480546 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.480517 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:44.758429 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:39:44.758337 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:39:44.843459 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:44.843433 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7"] Nov 02 20:39:44.846634 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:39:44.846605 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf7ccbf_aa69_4265_aab2_77ba15d34ca0.slice/crio-22bd20b4911f60211a15498bbe5932dc2d989d262e65a207b9fb6f231f5366fe WatchSource:0}: Error finding container 22bd20b4911f60211a15498bbe5932dc2d989d262e65a207b9fb6f231f5366fe: Status 404 returned error can't find the container with id 22bd20b4911f60211a15498bbe5932dc2d989d262e65a207b9fb6f231f5366fe Nov 02 20:39:45.133463 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:45.133425 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" event={"ID":"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0","Type":"ContainerStarted","Data":"ae92f77d4a6385998aff22579c597d57df3b4abfa43dd1220108fee22ed746be"} Nov 02 20:39:45.133640 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:45.133474 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" event={"ID":"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0","Type":"ContainerStarted","Data":"22bd20b4911f60211a15498bbe5932dc2d989d262e65a207b9fb6f231f5366fe"} Nov 02 20:39:45.238612 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:45.238554 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" podStartSLOduration=1.238538903 podStartE2EDuration="1.238538903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:39:45.238051005 +0000 UTC m=+4291.186706856" watchObservedRunningTime="2026-11-02 20:39:45.238538903 +0000 UTC m=+4291.187194801" Nov 02 20:39:46.140847 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:46.140807 4966 generic.go:356] "Generic (PLEG): container finished" podID="dbf7ccbf-aa69-4265-aab2-77ba15d34ca0" containerID="ae92f77d4a6385998aff22579c597d57df3b4abfa43dd1220108fee22ed746be" exitCode=0 Nov 02 20:39:46.141056 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:46.140899 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" event={"ID":"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0","Type":"ContainerDied","Data":"ae92f77d4a6385998aff22579c597d57df3b4abfa43dd1220108fee22ed746be"} Nov 02 20:39:47.281520 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.281491 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:47.377152 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.377112 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-kube-api-access-xkmtm\" (UniqueName: \"kubernetes.io/projected/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-kube-api-access-xkmtm\") pod \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " Nov 02 20:39:47.377373 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.377228 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-xks-ca\") pod \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " Nov 02 20:39:47.377373 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.377265 4966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-openshift-ca\") pod \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\" (UID: \"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0\") " Nov 02 20:39:47.377691 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.377658 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-openshift-ca" pod "dbf7ccbf-aa69-4265-aab2-77ba15d34ca0" (UID: "dbf7ccbf-aa69-4265-aab2-77ba15d34ca0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:47.379390 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.379370 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-xks-ca" pod "dbf7ccbf-aa69-4265-aab2-77ba15d34ca0" (UID: "dbf7ccbf-aa69-4265-aab2-77ba15d34ca0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:47.379498 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.379468 4966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-kube-api-access-xkmtm" pod "dbf7ccbf-aa69-4265-aab2-77ba15d34ca0" (UID: "dbf7ccbf-aa69-4265-aab2-77ba15d34ca0"). InnerVolumeSpecName "kube-api-access-xkmtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:47.478702 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.478666 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:47.478702 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.478698 4966 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:47.478893 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:47.478710 4966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmtm\" (UniqueName: \"kubernetes.io/projected/dbf7ccbf-aa69-4265-aab2-77ba15d34ca0-kube-api-access-xkmtm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:48.154645 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:48.154619 4966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" Nov 02 20:39:48.154840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:48.154621 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ss4d7" event={"ID":"dbf7ccbf-aa69-4265-aab2-77ba15d34ca0","Type":"ContainerDied","Data":"22bd20b4911f60211a15498bbe5932dc2d989d262e65a207b9fb6f231f5366fe"} Nov 02 20:39:48.154840 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:39:48.154740 4966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22bd20b4911f60211a15498bbe5932dc2d989d262e65a207b9fb6f231f5366fe" Nov 02 20:40:06.520432 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:06.520394 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5846d6f4ff-m9hwk_b60e0ac4-f6f1-47ed-b31f-fcf481449ba7/authorino/0.log" Nov 02 20:40:14.752583 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:40:14.752538 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:40:27.158283 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:27.158246 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pxv8z_fa3d028b-c7ed-4757-9161-6b55dc780ff8/manager/0.log" Nov 02 20:40:27.417469 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:27.417375 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4sl5m_b80d8188-2872-4626-8ee7-8177b778d47d/manager/0.log" Nov 02 20:40:27.717398 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:27.717319 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wcnxm_cb8b1d15-8458-4c37-8da8-b84182728c1c/manager/0.log" Nov 02 20:40:28.027846 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:28.027811 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddc6f7dd5-p4d2c_bcedd47a-6389-43d4-9765-b0a34bfe88f6/manager/0.log" Nov 02 20:40:28.673269 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:28.673238 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fnf5k_799aca84-787c-4e88-b7f1-fea3975ec850/manager/0.log" Nov 02 20:40:28.927740 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:28.927666 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rvzdb_332785e3-c94e-47bd-8c45-8440843708b4/proxy/0.log" Nov 02 20:40:30.208752 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:30.208722 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-2kq92_5cedfbf1-a097-441c-a67a-bd511d6e70c9/revoke-keys/0.log" Nov 02 20:40:30.412806 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:30.412780 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ss4d7_dbf7ccbf-aa69-4265-aab2-77ba15d34ca0/revoke-keys/0.log" Nov 02 20:40:30.622308 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:30.622279 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-67lp4_a9a2a485-aef1-49cb-b826-b1eef3dfcd70/revoke-keys/0.log" Nov 02 20:40:32.355595 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:32.355560 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5846d6f4ff-m9hwk_b60e0ac4-f6f1-47ed-b31f-fcf481449ba7/authorino/0.log" Nov 02 20:40:33.070000 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:33.069964 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-66g5j_5771e83c-8ce8-45d0-94d9-c9ef32ece870/manager/0.log" Nov 02 20:40:33.550983 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:33.550948 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-t2nvp_e11b4daa-eb53-4acf-b7e6-2a22b154d939/manager/0.log" Nov 02 20:40:34.136473 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:34.136432 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qv4jk_3e87633c-6b24-4c24-819d-ee74ce25ede3/manager/0.log" Nov 02 20:40:34.899196 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:34.899163 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b65854hbcw_9ec0732e-a572-4c2d-bc02-708dc2480f1c/istio-proxy/0.log" Nov 02 20:40:35.870714 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:35.870679 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-54rpb_c135774f-3f52-4d4c-81cb-aa2c86ea37d5/istio-proxy/1.log" Nov 02 20:40:36.132061 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:36.131980 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-5tkxm_539d4970-b4c7-4ecf-ab7c-be947313ce04/istio-proxy/0.log" Nov 02 20:40:44.753051 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:40:44.753011 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:40:46.356330 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:46.356281 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-98mww_895f69ba-a25f-44c4-ab3a-be99dddd93f1/istio-proxy/0.log" Nov 02 20:40:48.201044 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:48.201011 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm_8c852f75-6a1d-4521-a67f-20fd5db57355/istio-proxy/0.log" Nov 02 20:40:48.894981 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:48.894947 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9cf6bf74f-dljwv_596ee397-e505-475e-aa1f-6d7096d8312b/router/0.log" Nov 02 20:40:49.464601 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:49.464565 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-54rpb_c135774f-3f52-4d4c-81cb-aa2c86ea37d5/istio-proxy/0.log" Nov 02 20:40:52.509858 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:40:52.509818 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5846d6f4ff-m9hwk_b60e0ac4-f6f1-47ed-b31f-fcf481449ba7/authorino/0.log" Nov 02 20:41:01.790401 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:01.790289 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pxv8z_fa3d028b-c7ed-4757-9161-6b55dc780ff8/manager/0.log" Nov 02 20:41:02.050625 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:02.050545 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4sl5m_b80d8188-2872-4626-8ee7-8177b778d47d/manager/0.log" Nov 02 20:41:02.349938 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:02.349844 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wcnxm_cb8b1d15-8458-4c37-8da8-b84182728c1c/manager/0.log" Nov 02 20:41:02.625570 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:02.625469 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddc6f7dd5-p4d2c_bcedd47a-6389-43d4-9765-b0a34bfe88f6/manager/0.log" Nov 02 20:41:03.080997 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:03.080961 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fnf5k_799aca84-787c-4e88-b7f1-fea3975ec850/manager/0.log" Nov 02 20:41:03.323700 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:03.323669 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rvzdb_332785e3-c94e-47bd-8c45-8440843708b4/proxy/0.log" Nov 02 20:41:04.404385 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:04.404351 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-2kq92_5cedfbf1-a097-441c-a67a-bd511d6e70c9/revoke-keys/0.log" Nov 02 20:41:04.623417 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:04.623381 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ss4d7_dbf7ccbf-aa69-4265-aab2-77ba15d34ca0/revoke-keys/0.log" Nov 02 20:41:04.830058 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:04.830027 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-67lp4_a9a2a485-aef1-49cb-b826-b1eef3dfcd70/revoke-keys/0.log" Nov 02 20:41:14.751806 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:41:14.751764 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:41:16.177845 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:16.177804 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5846d6f4ff-m9hwk_b60e0ac4-f6f1-47ed-b31f-fcf481449ba7/authorino/0.log" Nov 02 20:41:16.935322 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:16.935254 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-66g5j_5771e83c-8ce8-45d0-94d9-c9ef32ece870/manager/0.log" Nov 02 20:41:17.402410 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:17.402378 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-t2nvp_e11b4daa-eb53-4acf-b7e6-2a22b154d939/manager/0.log" Nov 02 20:41:17.955097 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:17.955053 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qv4jk_3e87633c-6b24-4c24-819d-ee74ce25ede3/manager/0.log" Nov 02 20:41:18.557550 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:18.557515 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b65854hbcw_9ec0732e-a572-4c2d-bc02-708dc2480f1c/istio-proxy/0.log" Nov 02 20:41:19.575311 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:19.575262 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-54rpb_c135774f-3f52-4d4c-81cb-aa2c86ea37d5/istio-proxy/1.log" Nov 02 20:41:19.885538 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:19.885453 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-5tkxm_539d4970-b4c7-4ecf-ab7c-be947313ce04/istio-proxy/0.log" Nov 02 20:41:20.127040 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:20.127004 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-98mww_895f69ba-a25f-44c4-ab3a-be99dddd93f1/istio-proxy/0.log" Nov 02 20:41:21.910985 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:21.910949 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-8dtcm_8c852f75-6a1d-4521-a67f-20fd5db57355/istio-proxy/0.log" Nov 02 20:41:22.615080 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:41:22.615044 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9cf6bf74f-dljwv_596ee397-e505-475e-aa1f-6d7096d8312b/router/0.log" Nov 02 20:41:44.751166 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:41:44.751123 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:42:01.390759 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:01.390725 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddc6f7dd5-p4d2c_bcedd47a-6389-43d4-9765-b0a34bfe88f6/manager/0.log" Nov 02 20:42:01.928169 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:01.928137 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddc6f7dd5-p4d2c_bcedd47a-6389-43d4-9765-b0a34bfe88f6/manager/0.log" Nov 02 20:42:14.751171 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:42:14.751132 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:42:19.980604 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:19.980570 4966 scope.go:122] "RemoveContainer" containerID="9187d0e727c70d0225e207101785eae5151e36a5ab12a5b8bf01e1d3088cbb89" Nov 02 20:42:44.751131 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:42:44.751093 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:42:46.146653 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:46.146618 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddc6f7dd5-p4d2c_bcedd47a-6389-43d4-9765-b0a34bfe88f6/manager/0.log" Nov 02 20:42:46.677817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:46.677784 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddc6f7dd5-p4d2c_bcedd47a-6389-43d4-9765-b0a34bfe88f6/manager/0.log" Nov 02 20:42:47.965268 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:47.965234 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjlh5/must-gather-7qw2x"] Nov 02 20:42:47.965686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:47.965654 4966 cpu_manager.go:414] "containerMap: removing container" podUID="dbf7ccbf-aa69-4265-aab2-77ba15d34ca0" containerName="revoke-keys" Nov 02 20:42:47.965686 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:47.965666 4966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf7ccbf-aa69-4265-aab2-77ba15d34ca0" containerName="revoke-keys" Nov 02 20:42:47.965769 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:47.965761 4966 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf7ccbf-aa69-4265-aab2-77ba15d34ca0" containerName="revoke-keys" Nov 02 20:42:47.969114 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:47.969096 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.017498 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.017468 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjlh5\"/\"kube-root-ca.crt\"" Nov 02 20:42:48.063133 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.063094 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c54b245-7be9-42b4-b3ab-f1db168cec94-must-gather-output\") pod \"must-gather-7qw2x\" (UID: \"7c54b245-7be9-42b4-b3ab-f1db168cec94\") " pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.063387 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.063362 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48j65\" (UniqueName: \"kubernetes.io/projected/7c54b245-7be9-42b4-b3ab-f1db168cec94-kube-api-access-48j65\") pod \"must-gather-7qw2x\" (UID: \"7c54b245-7be9-42b4-b3ab-f1db168cec94\") " pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.063541 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.063517 4966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjlh5\"/\"openshift-service-ca.crt\"" Nov 02 20:42:48.063597 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.063569 4966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pjlh5\"/\"default-dockercfg-2k858\"" Nov 02 20:42:48.066092 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.066061 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/must-gather-7qw2x"] Nov 02 20:42:48.164896 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.164857 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48j65\" (UniqueName: \"kubernetes.io/projected/7c54b245-7be9-42b4-b3ab-f1db168cec94-kube-api-access-48j65\") pod \"must-gather-7qw2x\" (UID: \"7c54b245-7be9-42b4-b3ab-f1db168cec94\") " pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.165089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.165017 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c54b245-7be9-42b4-b3ab-f1db168cec94-must-gather-output\") pod \"must-gather-7qw2x\" (UID: \"7c54b245-7be9-42b4-b3ab-f1db168cec94\") " pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.165396 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.165378 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c54b245-7be9-42b4-b3ab-f1db168cec94-must-gather-output\") pod \"must-gather-7qw2x\" (UID: \"7c54b245-7be9-42b4-b3ab-f1db168cec94\") " pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.218149 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.218074 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48j65\" (UniqueName: \"kubernetes.io/projected/7c54b245-7be9-42b4-b3ab-f1db168cec94-kube-api-access-48j65\") pod \"must-gather-7qw2x\" (UID: \"7c54b245-7be9-42b4-b3ab-f1db168cec94\") " pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.284044 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.284008 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/must-gather-7qw2x" Nov 02 20:42:48.656974 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.656942 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/must-gather-7qw2x"] Nov 02 20:42:48.659633 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:42:48.659591 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c54b245_7be9_42b4_b3ab_f1db168cec94.slice/crio-fb78ce210370471ce5244574d9490683f251fa7bb96e8487d431962ec224fd8b WatchSource:0}: Error finding container fb78ce210370471ce5244574d9490683f251fa7bb96e8487d431962ec224fd8b: Status 404 returned error can't find the container with id fb78ce210370471ce5244574d9490683f251fa7bb96e8487d431962ec224fd8b Nov 02 20:42:48.661507 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:48.661487 4966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:42:49.187764 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:49.187727 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/must-gather-7qw2x" event={"ID":"7c54b245-7be9-42b4-b3ab-f1db168cec94","Type":"ContainerStarted","Data":"fb78ce210370471ce5244574d9490683f251fa7bb96e8487d431962ec224fd8b"} Nov 02 20:42:52.210972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:52.210882 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/must-gather-7qw2x" event={"ID":"7c54b245-7be9-42b4-b3ab-f1db168cec94","Type":"ContainerStarted","Data":"9d61864bad599dbf4408b2c40a5cfad1c2682bfc4d54cf281e747882319f70de"} Nov 02 20:42:52.210972 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:52.210928 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/must-gather-7qw2x" event={"ID":"7c54b245-7be9-42b4-b3ab-f1db168cec94","Type":"ContainerStarted","Data":"f5bcc3c7e5e15d03663aae77a8504685099900d94d5f2b3c554cb176ee5efaec"} Nov 02 20:42:52.317369 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:52.317288 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pjlh5/must-gather-7qw2x" podStartSLOduration=2.404097512 podStartE2EDuration="5.317276162s" totalImagesPullingTime="2.91317865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:52.314790639 +0000 UTC m=+4478.263446488" watchObservedRunningTime="2026-11-02 20:42:52.317276162 +0000 UTC m=+4478.265932001" Nov 02 20:42:58.056089 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:58.056053 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2dw9d_fea7ce1f-d9c4-40b4-a160-0f7788c416a6/global-pull-secret-syncer/0.log" Nov 02 20:42:59.804122 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:42:59.804090 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-52pkd_85cc6f7e-1889-40bb-a184-2138db4d29ba/connection-checker/0.log" Nov 02 20:43:00.199134 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:00.199059 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k8shl_84e7c05f-e63a-4829-ab28-c25bf61f9193/connection-checker/0.log" Nov 02 20:43:00.592054 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:00.592023 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-txxgt_a3ff325f-d384-407e-af38-2c876d9b686a/connection-checker/0.log" Nov 02 20:43:01.973377 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:01.973287 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xfpnh_d14e549d-addd-40f6-b6af-12582e1a028b/konnectivity-agent/0.log" Nov 02 20:43:02.753997 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:02.753950 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal_3ac4092c55873e071f156a04a17c98a5/haproxy/0.log" Nov 02 20:43:07.298694 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:07.298647 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5846d6f4ff-m9hwk_b60e0ac4-f6f1-47ed-b31f-fcf481449ba7/authorino/0.log" Nov 02 20:43:07.992543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:07.992504 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-66g5j_5771e83c-8ce8-45d0-94d9-c9ef32ece870/manager/0.log" Nov 02 20:43:08.420393 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.420274 4966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf"] Nov 02 20:43:08.427654 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.427617 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.507544 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.507508 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf"] Nov 02 20:43:08.577979 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.577940 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-proc\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.578227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.577996 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-podres\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.578227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.578130 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-sys\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.578227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.578171 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-lib-modules\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.578227 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.578200 4966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzp8q\" (UniqueName: \"kubernetes.io/projected/e9b16e53-80c8-478e-93b2-c6161d4e42ed-kube-api-access-hzp8q\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679565 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-proc\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679666 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679632 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-podres\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679673 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-sys\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679693 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-lib-modules\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679695 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-proc\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679719 4966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzp8q\" (UniqueName: \"kubernetes.io/projected/e9b16e53-80c8-478e-93b2-c6161d4e42ed-kube-api-access-hzp8q\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679764 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-podres\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679786 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-sys\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.679922 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.679874 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9b16e53-80c8-478e-93b2-c6161d4e42ed-lib-modules\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.733016 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.732980 4966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzp8q\" (UniqueName: \"kubernetes.io/projected/e9b16e53-80c8-478e-93b2-c6161d4e42ed-kube-api-access-hzp8q\") pod \"perf-node-gather-daemonset-j2dpf\" (UID: \"e9b16e53-80c8-478e-93b2-c6161d4e42ed\") " pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.744817 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.744761 4966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:08.894885 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:08.894850 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-t2nvp_e11b4daa-eb53-4acf-b7e6-2a22b154d939/manager/0.log" Nov 02 20:43:09.115741 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:09.115709 4966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf"] Nov 02 20:43:09.117334 ip-10-0-132-174 kubenswrapper[4966]: W1102 20:43:09.117272 4966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode9b16e53_80c8_478e_93b2_c6161d4e42ed.slice/crio-c31059b9a7f473dc68918fbd1b9b17309051c616120e228972207320aae72790 WatchSource:0}: Error finding container c31059b9a7f473dc68918fbd1b9b17309051c616120e228972207320aae72790: Status 404 returned error can't find the container with id c31059b9a7f473dc68918fbd1b9b17309051c616120e228972207320aae72790 Nov 02 20:43:09.389543 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:09.389453 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" event={"ID":"e9b16e53-80c8-478e-93b2-c6161d4e42ed","Type":"ContainerStarted","Data":"75f2d9b84cd411890be549a55b29602856cae04e81cbd6cd6ccbb08df0c7a865"} Nov 02 20:43:09.389754 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:09.389565 4966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" event={"ID":"e9b16e53-80c8-478e-93b2-c6161d4e42ed","Type":"ContainerStarted","Data":"c31059b9a7f473dc68918fbd1b9b17309051c616120e228972207320aae72790"} Nov 02 20:43:09.390983 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:09.390558 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:09.495864 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:09.495810 4966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" podStartSLOduration=1.49579582 podStartE2EDuration="1.49579582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:09.494615267 +0000 UTC m=+4495.443271097" watchObservedRunningTime="2026-11-02 20:43:09.49579582 +0000 UTC m=+4495.444451658" Nov 02 20:43:09.692681 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:09.692550 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qv4jk_3e87633c-6b24-4c24-819d-ee74ce25ede3/manager/0.log" Nov 02 20:43:14.759583 ip-10-0-132-174 kubenswrapper[4966]: E1102 20:43:14.759538 4966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fb2w5" containerName="extractor" Nov 02 20:43:14.828616 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:14.828585 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ffh8p_2939006c-2a64-4684-968e-dc69c92c23ea/cluster-monitoring-operator/0.log" Nov 02 20:43:15.337203 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:15.337169 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:43:15.358011 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:15.357983 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:43:15.363993 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:15.363968 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvz4fn_b8c4ec3c-5fe5-47b4-8630-2520bd7bab6a/extract/0.log" Nov 02 20:43:15.378293 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:15.378271 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdcml_1804f9cc-5858-4cbb-a623-af98ee3234c2/console-operator/3.log" Nov 02 20:43:16.413294 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:16.413261 4966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pjlh5/perf-node-gather-daemonset-j2dpf" Nov 02 20:43:19.515028 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:19.514993 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5ppq_62c8a1d4-ef6e-45ea-b0ee-3354a556b978/node-exporter/0.log" Nov 02 20:43:19.908624 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:19.908539 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5ppq_62c8a1d4-ef6e-45ea-b0ee-3354a556b978/kube-rbac-proxy/0.log" Nov 02 20:43:20.294751 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:20.294725 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5ppq_62c8a1d4-ef6e-45ea-b0ee-3354a556b978/init-textfile/0.log" Nov 02 20:43:30.682927 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:30.682882 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b65854hbcw_9ec0732e-a572-4c2d-bc02-708dc2480f1c/istio-proxy/0.log" Nov 02 20:43:31.586614 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:31.586572 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-fvrzl_7bf9d232-57b4-42b2-9919-b03b3170fe02/networking-console-plugin/0.log" Nov 02 20:43:31.808357 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:31.808292 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-54rpb_c135774f-3f52-4d4c-81cb-aa2c86ea37d5/istio-proxy/0.log" Nov 02 20:43:31.809872 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:31.809849 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-54rpb_c135774f-3f52-4d4c-81cb-aa2c86ea37d5/istio-proxy/1.log" Nov 02 20:43:31.959510 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:31.959429 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-5tkxm_539d4970-b4c7-4ecf-ab7c-be947313ce04/istio-proxy/0.log" Nov 02 20:43:32.155848 ip-10-0-132-174 kubenswrapper[4966]: I1102 20:43:32.155802 4966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-98mww_895f69ba-a25f-44c4-ab3a-be99dddd93f1/istio-proxy/0.log"